Causal Language Modeling (Causal LM)

Causal language modeling is training (or running) a model to predict the next token given only the tokens to its left. The model is unidirectional: it never attends to future positions. That is the standard autoregressive / decoder-only setup.

Example:

Input: “The weather is” → Predict: “sunny”

Decoder-only LLMs are causal language models:

  • GPT-2 / GPT-3 / GPT-4-class models
  • Gemma
  • LLaMA
  • Falcon
  • Qwen

A causal attention mask zeros out (or blocks) future positions so training can still run in parallel over the sequence while preserving left-context-only predictions. Contrast with a masked model (BERT-style), which predicts a blank in the middle using both sides.

Related: next-token prediction, pretraining. For adapting an already-trained causal LM to a private corpus, see continued pretraining.

← Back to AI Glossary