Masked Model
Also known as: Masked Language Model, MLM
A masked model (masked language model, MLM) is trained to recover deliberately hidden spans of the input using bidirectional context. BERT-style encoders replace random tokens with a mask token and predict the originals. Classic self-supervised pretraining.
MLM vs causal LM
| Masked model (MLM) | Causal LM | |
|---|---|---|
| Context | Left and right | Left only |
| Typical stack | Encoder (BERT-style) | Decoder-only (GPT-style) |
| Training target | Recover masked positions | Next-token prediction |
| Strength | NLU, embeddings, NER | Open-ended generation |
| Generation | Not the default use | Native left-to-right text |
See causal language model for the unidirectional case most chat LLMs use.
Why it matters
- Strong for understanding tasks: classification, NER, embeddings
- Sees left and right context, unlike causal LMs
- Less natural for open-ended left-to-right generation
How it shows up
Training objective: minimize cross-entropy on masked positions only. At inference, you typically fine-tune or pool hidden states rather than “unmask” free-form prose. Modern LLMs are mostly decoder-only / causal; masked (and encoder-decoder) models still matter for retrieval encoders, rerankers, and classical NLU pipelines.
Related: span corruption (T5) is a close cousin that corrupts contiguous spans instead of single tokens.