Causal Language Modeling (Causal LM)
Causal language modeling is training (or running) a model to predict the next token given only the tokens to its left. The model is unidirectional: it never attends to future positions. That is the standard autoregressive / decoder-only setup.
Example:
Input: “The weather is” → Predict: “sunny”
Decoder-only LLMs are causal language models:
- GPT-2 / GPT-3 / GPT-4-class models
- Gemma
- LLaMA
- Falcon
- Qwen
A causal attention mask zeros out (or blocks) future positions so training can still run in parallel over the sequence while preserving left-context-only predictions. Contrast with a masked model (BERT-style), which predicts a blank in the middle using both sides.
Related: next-token prediction, pretraining. For adapting an already-trained causal LM to a private corpus, see continued pretraining.