Synthetic Data
Synthetic data is generated by models or simulators for training or eval—e.g., LLM-written instruction pairs, rewritten documents, or simulated tool traces.
It scales labeling when human annotation is slow or expensive, and it can target rare edge cases. The risk: it amplifies teacher biases, style quirks, and hallucinations. A student trained only on a teacher’s outputs may never exceed the teacher on factual ground truth.
Hygiene
- Filter for format, toxicity, and factuality where possible
- Mix synthetic with real domain text when available
- Hold out real eval sets; never grade only on synthetic twins of the train set
- Prefer diverse teachers and templates over one prompt repeated forever
Synthetic SFT corpora and knowledge distillation pipelines are closely related: both treat a stronger model as a noisy labeler.