Pretraining

Pretraining is the large-scale initial training phase where a model learns general representations from massive unlabeled (or weakly labeled) data — e.g., next-token prediction or masked language modeling on web-scale text.

Downstream performance usually depends more on pretraining data mix and scale than on any single fine-tuning trick. Continued pretraining is a smaller, targeted second pretraining stage on your corpus.

← Back to AI Glossary