A decoder-only model is a transformer built from one stack of decoder layers that generates text left to right, using causal attention so each position sees only earlier positions. There is no separate encoder: the prompt and the generated response live in the same sequence. GPT-style LLMs, including most modern chat models, use this architecture.
Key takeaways
- One stack of layers handles both understanding the prompt and generating the answer.
- Causal masking prevents any position from seeing future tokens.
- Training is simple: next-token prediction on raw text.
- The design scales well and supports in-context learning.
- It has replaced encoder-decoder designs for general-purpose LLMs.
A decoder-only model explained in simple terms
Imagine writing a document where you can read everything you have written so far but nothing ahead. Your prompt is the first part of the document, and the answer is what you keep writing. There is no separate “reading department” and “writing department”: one process does both, and the prompt simply sets the scene.
How it works
- The prompt is tokenised and embedded.
- Layers of causal self-attention and feed-forward networks process the sequence, with each position attending only to itself and earlier positions.
- The final layer produces logits over the whole vocabulary for the next token.
- A token is sampled, appended to the sequence, and the process repeats.
During inference, the model caches keys and values so it does not reprocess the prompt for every new token.
Decoder-only vs other architectures
| Aspect | Decoder-only | Encoder-only | Encoder-decoder |
|---|---|---|---|
| Attention over input | Causal (left to right) | Bidirectional | Bidirectional input, causal output |
| Produces text | Yes | No | Yes |
| Training objective | Next-token prediction | Masked language modelling | Sequence to sequence |
| Examples | GPT-style LLMs | BERT | T5, BART |
| Best at | Open-ended generation, chat, code | Classification, embeddings | Translation, summarisation |
Why decoder-only models became dominant
- Simplicity: one stack, one objective, one loss.
- Data efficiency at scale: any text can be training data; every token provides a prediction target.
- Task flexibility: most tasks can be phrased as text continuation, so no task-specific heads are needed.
- In-context learning: examples in the prompt steer behaviour without retraining.
- Serving efficiency: the KV cache makes incremental generation practical.
A worked example
Prompt: “Translate to Hindi: Good morning.” A decoder-only model treats this as text to continue. Its causal attention lets the tokens of “Good morning” inform what comes next, and instruction tuning has taught it that such a prompt should be followed by a Hindi translation rather than more English. One architecture handles translation, summarisation and chat, purely by continuing text appropriately.
Trade-offs
Because attention is causal, earlier tokens cannot see later ones, so the model’s representation of the prompt is one-directional. Encoder-only models can be stronger for pure understanding tasks such as classification and embeddings at a given size. In practice, the flexibility and scale of decoder-only models usually outweighs this.
Limitations
- Generation is sequential, limiting speed.
- No bidirectional view of the prompt.
- Long prompts consume context window and memory.
- Errors early in generation propagate forward.