Synthetic Data

Synthetic data is generated by models or simulators for training or eval—e.g., LLM-written instruction pairs, rewritten documents, or simulated tool traces.

It scales labeling when human annotation is slow or expensive, and it can target rare edge cases. The risk: it amplifies teacher biases, style quirks, and hallucinations. A student trained only on a teacher’s outputs may never exceed the teacher on factual ground truth.

Hygiene

  • Filter for format, toxicity, and factuality where possible
  • Mix synthetic with real domain text when available
  • Hold out real eval sets; never grade only on synthetic twins of the train set
  • Prefer diverse teachers and templates over one prompt repeated forever

Synthetic SFT corpora and knowledge distillation pipelines are closely related: both treat a stronger model as a noisy labeler.

← Back to AI Glossary