A causal language model (CLM) predicts each token using only the tokens that came before it. “Causal” here means the past can influence the future but not the other way round. This left-to-right constraint, enforced by a causal mask in self-attention, is what allows a model to generate text one token at a time. Nearly all generative LLMs are causal language models.
Key takeaways
- Causal models are also called autoregressive or left-to-right language models.
- A causal mask blocks attention to future positions during training and inference.
- The training objective is next-token prediction.
- Every token in the training text provides a learning signal.
- The alternative, masked language modelling, sees both directions but does not generate fluently.
A causal language model explained in simple terms
Think of reading a novel one word at a time with a card covering the rest of the page. At each point you can only use what you have already read to guess the next word. Training a causal language model is exactly this exercise, repeated across trillions of words until its guesses become very good.
How causal masking works
During training, the whole sequence is processed in parallel for speed. Without protection, position 5 could simply look ahead at position 6 and copy the answer. The causal mask prevents this by setting attention scores for future positions to negative infinity before softmax, making their weights zero.
Worked example with the sentence “AI models predict text”:
| Position | Can attend to | Must predict |
|---|---|---|
| 1: AI | AI | models |
| 2: models | AI, models | predict |
| 3: predict | AI, models, predict | text |
One pass through the sequence produces three separate training signals, all computed at once.
Causal LM vs masked LM
| Aspect | Causal LM | Masked LM |
|---|---|---|
| Context | Only previous tokens | Tokens on both sides |
| Objective | Predict the next token | Predict hidden tokens |
| Generation | Natural and fluent | Not designed for it |
| Understanding tasks | Good | Often stronger at the same size |
| Examples | GPT-style models, Llama | BERT, RoBERTa |
Why causal models dominate
A causal objective matches how text is produced, so the same model can chat, write, translate and code. It also uses data efficiently: every token is a prediction target, whereas masked models only learn from the small share of tokens they hide. And causal models support in-context learning, where examples inside the prompt steer behaviour.
Prefix and hybrid variants
Some architectures allow bidirectional attention over the prompt and causal attention over the generated part, sometimes called prefix language modelling. This gives better understanding of the input while keeping generation causal. Fill-in-the-middle training, common in code models, teaches models to complete a gap by rearranging the text so the task remains causal.
Limitations
- The model cannot revise earlier tokens once generated.
- Representations of the prompt are one-directional.
- Sequential generation limits speed.
- Early mistakes affect everything that follows.