An attention mechanism is a technique that lets a neural network decide which parts of its input matter most for each part of its output. Rather than compressing everything into one fixed summary, the model scores the relevance of each input element and combines them accordingly. Attention is the core idea behind transformers and therefore behind modern LLMs.
Key takeaways
- Attention computes relevance scores, then takes a weighted mix of information.
- It was introduced for machine translation in 2014 by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio.
- The 2017 transformer paper showed attention alone could replace recurrence.
- Main forms: self-attention, cross-attention and causal (masked) attention.
- Attention makes long-range connections direct instead of passing through many steps.
An attention mechanism explained in simple terms
When translating a long sentence, you do not hold every word in mind equally. As you write each word, you glance back at the few source words that matter for it. Attention gives a model the same ability: for each output step, it looks back over the input and weighs what is relevant right now.
Why attention was invented
Early translation models used an RNN encoder to squeeze a whole sentence into one fixed vector, then a decoder unpacked it. Long sentences overflowed that bottleneck. Attention removed it: the decoder could look at every encoder state directly, weighted by relevance.
How attention works
- Create a query representing what the current position needs.
- Compare it with a key for every input position to get scores.
- Turn scores into weights with softmax, so they sum to 1.
- Take a weighted sum of the values attached to each position.
- Use the result as context for the current position.
Types of attention
| Type | What attends to what | Used in |
|---|---|---|
| Self-attention | Positions within the same sequence | Transformer encoders and decoders |
| Cross-attention | Decoder positions attend to encoder output | Encoder-decoder models, multimodal models |
| Causal (masked) | Each position attends only to earlier positions | Decoder-only models |
| Additive (Bahdanau) | Scores from a small neural network | 2014-era translation models |
| Scaled dot-product | Scores from dot products divided by the square root of the dimension | Transformers |
| Multi-head | Several attention operations in parallel | All modern transformers |
A worked example
Input: “The keys to the cabinet are on the table.” When processing “are”, attention scores might be high for “keys” (0.6), moderate for “cabinet” (0.1) and low elsewhere. The weighted mix tells the model the subject is plural, so “are” fits. No fixed rule was written; the weights are computed from learned representations.
Efficient attention
Standard attention compares every position with every other, so cost grows with the square of sequence length. Research on efficient variants includes FlashAttention (2022) for faster exact attention, sliding-window and sparse attention, and grouped-query attention to shrink the memory used during inference.
Limitations
- Quadratic cost limits very long context windows.
- Attention weights are not a complete explanation of model behaviour.
- Models can attend to irrelevant tokens, especially in long inputs.