A transformer is a neural network architecture that uses a mechanism called self-attention to relate every part of an input to every other part at the same time. Introduced by Google researchers in the 2017 paper “Attention Is All You Need”, it replaced RNNs for most language tasks and now underpins nearly all large language models, as well as many image, audio and video models.
Key takeaways
- Self-attention lets each token look at all other tokens to understand context.
- Transformers process sequences in parallel, making training on GPUs far faster than RNNs.
- Core parts: embeddings, positional information, multi-head attention, feed-forward layers, residual connections and normalisation.
- Variants include encoder-only (BERT), decoder-only (GPT) and encoder-decoder (T5).
- Attention cost grows quickly with sequence length, which limits the context window.
A transformer explained in simple terms
Read this sentence: “The trophy didn’t fit in the suitcase because it was too big.” To know what “it” means, you look back at “trophy” and “suitcase” and decide. Attention is the transformer’s way of doing this. For every word, it asks: which other words matter for understanding this one, and how much? It does this for all words at once, in many layers, building up a detailed understanding of the whole text.
How self-attention works
Each token is turned into three vectors:
- Query: what this token is looking for.
- Key: what this token offers.
- Value: the information this token passes on.
The model compares each token’s query with every token’s key to get attention scores, turns the scores into weights with softmax, and takes a weighted mix of the values. In the example above, the query for “it” matches strongly with the key for “trophy”, so “trophy” contributes most to how “it” is understood.
Multi-head attention runs several attention operations in parallel. Different heads can track different relationships, such as grammar, meaning or references.
The parts of a transformer block
| Component | Role |
|---|---|
| Token embeddings | Turn tokens into vectors |
| Positional information | Tells the model word order, since attention alone ignores order |
| Multi-head self-attention | Mixes information across tokens |
| Feed-forward network | Processes each token’s representation further |
| Residual connections | Add each layer’s input to its output, easing training of deep stacks |
| Layer normalisation | Stabilises training |
Large models stack dozens of these blocks.
Transformer variants
| Variant | How it works | Best at | Examples |
|---|---|---|---|
| Encoder-only | Reads the full input in both directions | Understanding, classification, embeddings | BERT |
| Decoder-only | Generates left to right with a causal mask | Text generation, chat, code | GPT-style LLMs |
| Encoder-decoder | Encoder reads input; decoder generates output | Translation, summarisation | Original transformer, T5 |
| Vision transformer | Treats image patches like tokens | Image recognition | ViT |
Why transformers won
- Parallel training: unlike RNNs, all positions are processed together.
- Long-range context: any token can attend directly to any other.
- Scalability: performance kept improving as models, data and compute grew.
- Flexibility: the same design works for text, images, audio and more.
Limitations
- Standard attention cost grows with the square of sequence length, making long contexts expensive.
- Memory for the attention cache grows during inference.
- Large transformers need huge data and compute.
- Internal workings are hard to interpret.
Researchers address these with efficient attention methods, mixture-of-experts designs and alternatives such as state space models.