An encoder-decoder model, also called a sequence-to-sequence (seq2seq) model, uses two components: an encoder that reads and represents the input, and a decoder that generates the output. The decoder looks back at the encoder’s representation through cross-attention while producing each token. The original 2017 transformer was an encoder-decoder built for translation.
Key takeaways
- The encoder reads the whole input at once, in both directions.
- The decoder generates output left to right, attending to both its own output and the encoder’s.
- Cross-attention is the bridge between the two.
- Examples include the original transformer, T5, BART and many translation and speech models.
- Most general-purpose LLMs today are decoder-only instead.
An encoder-decoder explained in simple terms
Think of an interpreter at a conference. First they listen to the full sentence in Hindi and build an understanding of it (encoding). Then they speak the English sentence word by word, checking back against their understanding as they go (decoding with cross-attention). The two jobs are separate but connected.
How it works
- Encoding: the input sequence passes through encoder layers with bidirectional self-attention, producing one representation per input token.
- Decoding starts: the decoder begins with a start token.
- Causal self-attention: the decoder attends to the tokens it has generated so far.
- Cross-attention: the decoder attends to the encoder’s representations, choosing which parts of the input matter for the current output token.
- Generate: predict the next token, append it, repeat until an end token.
Encoder-decoder vs encoder-only vs decoder-only
| Architecture | Input handling | Output | Typical use |
|---|---|---|---|
| Encoder-only | Bidirectional | Representations, labels | Classification, embeddings (BERT) |
| Decoder-only | Left to right | Generated text | Chat, general LLMs (GPT-style) |
| Encoder-decoder | Bidirectional input, generated output | Generated text | Translation, summarisation (T5, BART) |
Where encoder-decoder models shine
- Translation: the input and output are different languages of different lengths.
- Summarisation: the input is long, the output short.
- Speech recognition: audio in, text out.
- Text-to-speech and audio: text in, audio features out.
- Structured transformations: converting one format into another.
The separation is useful when the input must be fully understood before output begins, and when the input is processed once but referenced repeatedly.
A worked example
Translating “The meeting has been postponed to Friday” into Hindi: the encoder builds a representation of all eight words at once, with full context in both directions. As the decoder writes each Hindi word, cross-attention lets it focus on the relevant English words, such as attending to “Friday” when producing the Hindi word for Friday, even though word order differs between the languages.
Why decoder-only models took over
Decoder-only models are simpler to train at scale, need only one stack of layers, and can treat any task as text continuation. With instruction tuning, they handle translation and summarisation well without a separate encoder. Encoder-decoder models remain strong in specialised tasks, especially translation and speech, and in settings where inputs are long and outputs short.
Limitations
- More complex than a single stack: two sets of layers plus cross-attention.
- Less flexible for open-ended chat.
- Fewer very large, general-purpose encoder-decoder LLMs are available.