A decoder is a neural network component that turns an internal numeric representation into output, such as text, an image or audio. In language models, a decoder generates text one token at a time, each step predicting the next token from everything produced so far. GPT-style large language models are built almost entirely from decoder layers.
Key takeaways
- A decoder produces output; an encoder produces representations.
- Text decoders are autoregressive: they generate step by step, feeding each new token back in.
- Decoder-only transformers power most modern chat and writing models.
- Decoding strategies such as greedy, beam search and temperature sampling shape the output.
- In images, decoders expand compact representations back into pixels.
A decoder explained in simple terms
Think of a translator who has fully understood a sentence in one language (the encoder’s job) and now speaks it aloud in another language, word by word. Each word they choose depends on the meaning they hold and the words they have already said. A decoder works the same way. It holds a representation of what it needs to say and builds the output one piece at a time.
How a text decoder works
- Start with a prompt or a start token.
- Process the tokens so far through decoder layers. A causal mask stops each position from looking at future tokens.
- Produce a probability for every possible next token.
- Choose one token using a decoding strategy.
- Add it to the sequence and repeat until an end token or length limit.
This loop is why longer answers take longer to produce, and why AI inference cost grows with output length.
A worked example: choosing the next word
Prompt: “The capital of France is”. The decoder outputs probabilities such as: Paris 0.92, a 0.03, located 0.02, and many tiny values. With greedy decoding, it picks “Paris”. With temperature sampling, it usually picks “Paris” but occasionally something else; higher temperature makes unusual choices more likely. After “Paris”, the model runs again to choose the next token, perhaps “.”.
Decoding strategies
| Strategy | How it chooses | Best for |
|---|---|---|
| Greedy | Always the most likely token | Short factual answers |
| Beam search | Keeps several candidate sequences, picks the best overall | Translation, summarisation |
| Temperature sampling | Samples from probabilities, scaled by temperature | Creative writing, variety |
| Top-k / top-p (nucleus) | Samples only from the most likely tokens | Balanced, fluent generation |
Decoder-only, encoder-only and encoder-decoder
| Architecture | Strength | Examples |
|---|---|---|
| Decoder-only | Open-ended generation, chat, code | GPT-style models and most modern LLMs |
| Encoder-only | Understanding and embeddings | BERT |
| Encoder-decoder | Transforming one sequence into another | Translation models, T5 |
In an encoder-decoder model, the decoder uses cross-attention to look at the encoder’s representation of the input while generating. Decoder-only models have no separate encoder; the prompt and the output share one sequence. This simple design scales well, which is one reason decoder-only models dominate large language modelling.
Decoders beyond text
- In an autoencoder, the decoder reconstructs the original input from a compressed code.
- In image generators, including diffusion models that work in a compressed “latent” space, a decoder turns the final latent back into a full image.
- In text-to-speech, decoders produce audio features or waveforms.
Limitations
- Autoregressive generation is sequential, so long outputs are slow.
- Early mistakes carry forward, because each token depends on previous ones.
- A decoder can produce fluent but false statements, known as AI hallucination.
- Output quality depends heavily on decoding settings.