Multi-head attention runs several attention operations in parallel, each with its own learned projections, and combines their results. Each “head” can focus on a different kind of relationship, such as grammatical agreement, references between words or topic similarity. Their outputs are concatenated and passed through a final linear layer.
Key takeaways
- One attention operation captures one pattern of relevance; several heads capture several.
- Each head works in a smaller dimension, so total cost is similar to one full-size head.
- The original 2017 transformer used 8 heads with a model dimension of 512.
- Modern efficiency variants include multi-query attention (MQA) and grouped-query attention (GQA).
- Heads are combined by concatenation followed by a learned output projection.
Multi-head attention explained in simple terms
Ask one expert to review a contract and you get one perspective. Ask a lawyer, an accountant and a project manager to review it in parallel and you get three different sets of observations, which you then merge. Multi-head attention is that panel of reviewers, and each head learns for itself what to look for.
How it works
- Split the work into h heads.
- For each head, project the input into smaller queries, keys and values using its own learned matrices.
- Run scaled dot-product attention in each head independently.
- Concatenate all head outputs.
- Pass the result through a final linear layer, W_O.
Dimensions: a worked example
Take a model dimension of 512 with 8 heads, as in the original transformer.
- Each head’s key and value dimension: 512 ÷ 8 = 64.
- Each head attends over the sequence using 64-dimensional vectors.
- Concatenating 8 × 64 gives back 512 before the output projection.
So multi-head attention divides the representation rather than multiplying the cost. A model with a dimension of 4,096 and 32 heads gives each head 128 dimensions.
What different heads learn
Studies of trained transformers have found heads with recognisable jobs: attending to the previous token, tracking syntactic relations, matching a word to its earlier occurrence, or copying patterns seen earlier in the text (often called induction heads). Many heads are less interpretable, and some can be removed with little loss, as research on head pruning has shown.
Efficiency variants
| Variant | Idea | Benefit |
|---|---|---|
| Multi-head attention (MHA) | Every head has its own keys and values | Most expressive |
| Multi-query attention (MQA) | All heads share one set of keys and values | Much smaller KV cache, faster inference |
| Grouped-query attention (GQA) | Heads share keys and values in groups | Balance of quality and speed; widely used in recent LLMs |
These variants mainly reduce the key-value memory that grows with context length during generation.
Limitations
- More heads do not always help; capacity per head shrinks.
- Some heads are redundant and can be pruned.
- Head behaviour is only partly interpretable.
- Still carries attention’s quadratic cost in sequence length.