Masked language modeling (MLM) is a training objective in which random tokens in a text are hidden and the model learns to predict them using the words on both sides. Because the model sees left and right context, it builds rich bidirectional representations. MLM is the objective behind BERT (2018) and many encoder models used for classification and embeddings.
Key takeaways
- MLM is a form of self-supervised learning: the text provides its own labels.
- BERT masked about 15% of tokens in each sequence.
- Bidirectional context makes MLM strong for understanding tasks.
- It is not designed for text generation, unlike causal language modelling.
- Variants include whole-word masking and span masking.
Masked language modeling explained in simple terms
It is a fill-in-the-blanks exercise. Take the sentence “The train to Mumbai leaves at ___ o’clock” and the model must guess the missing word. To do well, it must understand grammar, context and meaning from both sides of the gap. Repeat across billions of sentences and the model develops a deep sense of how language works.
How MLM training works
- Take a text sequence from the training data.
- Randomly select about 15% of the tokens.
- Replace them according to BERT’s scheme: 80% become a special [MASK] token, 10% become a random token and 10% are left unchanged.
- The model predicts the original token at each selected position.
- Cross-entropy loss measures the error only at those positions.
The 80/10/10 rule exists because the [MASK] token never appears when the model is later used on real text. Mixing in random and unchanged tokens forces the model to build good representations everywhere, not just at masks.
A worked example
Input: “QueryFeedz publishes articles about [MASK] and startups.”
The model considers the surrounding words. “Publishes articles” suggests a topic noun; “and startups” suggests a related domain. It may predict “technology” with high probability, “business” next, and unrelated words with very low probability. The error between its prediction and the true word updates the weights.
MLM vs causal language modelling
| Aspect | Masked LM | Causal LM |
|---|---|---|
| Context available | Both sides | Left only |
| Tokens predicted per sequence | About 15% | Every token |
| Data efficiency per pass | Lower | Higher |
| Text generation | Weak | Strong |
| Typical use | Classification, search, embeddings | Chat, writing, code |
| Examples | BERT, RoBERTa, DeBERTa | GPT-style LLMs |
Variants
- Whole-word masking: hide all subword pieces of a word together, which makes the task harder and more realistic.
- Span masking: hide contiguous spans, as in SpanBERT and T5’s approach.
- Replaced token detection: ELECTRA trains the model to spot which tokens were replaced, learning from every position.
- Masked image modelling: the same idea applied to image patches.
Where MLM models are still used
Encoder models trained with MLM remain popular for text classification, named entity recognition, reranking search results and producing sentence embeddings. They are small, fast and cheap compared with generative LLMs, which matters at high volume.
Limitations
- Cannot generate fluent text on its own.
- Learns from only a fraction of tokens per pass, so training is less data-efficient.
- A mismatch exists between training, which uses masks, and real use, which does not.
- Less able to follow instructions than modern instruction-tuned LLMs.