Self-attention is a mechanism in which every position in a sequence attends to every other position in the same sequence. Each token produces a query, a key and a value. Queries are matched against keys to produce relevance scores, and the scores are used to take a weighted mix of values. Self-attention is the core operation inside every transformer layer.
Key takeaways
- “Self” means the sequence attends to itself, rather than to a separate input.
- The formula is: softmax(QKᵀ ÷ √dₖ) × V.
- Dividing by the square root of the key dimension keeps scores in a stable range.
- Every position is computed in parallel, which makes training fast on GPUs.
- Causal self-attention masks future positions so generation stays left to right.
Self-attention explained in simple terms
Imagine everyone in a meeting writing a short note saying what they need (the query) and what they can offer (the key), plus the actual information they hold (the value). Each person reads all the offers, decides who is most relevant to their need, and combines those people’s information. Do this for everyone at once, and every participant ends up better informed about the whole discussion. That is one self-attention layer.
Queries, keys and values
| Element | Meaning | How it is made |
|---|---|---|
| Query (Q) | What this token is looking for | Token vector × learned matrix W_Q |
| Key (K) | What this token offers | Token vector × learned matrix W_K |
| Value (V) | The information passed on | Token vector × learned matrix W_V |
How self-attention is computed
- Turn each token’s embedding into Q, K and V vectors.
- Compute scores: the dot product of each query with every key.
- Scale: divide the scores by √dₖ, where dₖ is the key dimension.
- Mask (if causal): set scores for future positions to negative infinity.
- Softmax: convert scores into weights that sum to 1.
- Output: multiply weights by values and sum.
A worked example
Sentence: “The bank raised rates.” When processing “bank”, its query is compared with all keys. Suppose scaled scores are: The 0.2, bank 1.5, raised 2.1, rates 1.8. Softmax turns these into weights of roughly 0.07, 0.25, 0.37 and 0.31. The output for “bank” is dominated by “raised” and “rates”, so the model’s representation of “bank” leans towards the financial meaning rather than a riverbank. The next layer builds on that.
Why scale by √dₖ?
With large dimensions, dot products can become very large, pushing softmax into a region where one weight is near 1 and gradients almost vanish. Dividing by √dₖ keeps the scores in a moderate range so training stays stable.
Self-attention vs recurrence
| Aspect | Self-attention | RNN |
|---|---|---|
| Path between distant tokens | One step | Many steps |
| Parallel training | Yes | No |
| Cost per layer | Grows with the square of length | Grows linearly |
| Memory during generation | Grows with context (KV cache) | Constant |
Limitations
- Quadratic compute and memory in sequence length.
- Needs positional encoding, since attention alone ignores order.
- A single attention operation captures one pattern, which is why multi-head attention is used.