Masked Model

Also known as: Masked Language Model, MLM

A masked model (masked language model, MLM) is trained to recover deliberately hidden spans of the input using bidirectional context. BERT-style encoders replace random tokens with a mask token and predict the originals. Classic self-supervised pretraining.

MLM vs causal LM

  Masked model (MLM) Causal LM
Context Left and right Left only
Typical stack Encoder (BERT-style) Decoder-only (GPT-style)
Training target Recover masked positions Next-token prediction
Strength NLU, embeddings, NER Open-ended generation
Generation Not the default use Native left-to-right text

See causal language model for the unidirectional case most chat LLMs use.

Why it matters

  • Strong for understanding tasks: classification, NER, embeddings
  • Sees left and right context, unlike causal LMs
  • Less natural for open-ended left-to-right generation

How it shows up

Training objective: minimize cross-entropy on masked positions only. At inference, you typically fine-tune or pool hidden states rather than “unmask” free-form prose. Modern LLMs are mostly decoder-only / causal; masked (and encoder-decoder) models still matter for retrieval encoders, rerankers, and classical NLU pipelines.

Related: span corruption (T5) is a close cousin that corrupts contiguous spans instead of single tokens.

← Back to AI Glossary