Positional encoding is the method transformers use to tell the model where each token sits in a sequence. Self-attention compares tokens with each other but has no built-in sense of order, so without positional information “the dog bit the man” and “the man bit the dog” would look identical. Positional encoding injects order into the model.
Key takeaways
- Attention is order-blind, so position must be added explicitly.
- The original transformer used fixed sinusoidal encodings added to token embeddings.
- Later models used learned absolute positions, then relative positions.
- Most recent LLMs use rotary positional embedding (RoPE) or similar methods.
- Positional methods strongly affect how well a model handles long context.
Positional encoding explained in simple terms
Imagine receiving a sentence as a bag of loose word cards. You can see which words are present but not their order. Positional encoding is like stamping a position marker on each card before dropping it in the bag, so the reader can reconstruct the sequence.
Sinusoidal positional encoding
The 2017 transformer paper used fixed sine and cosine waves of different frequencies. Each position gets a unique pattern across the embedding dimensions, added to the token embedding. Because it is a formula rather than learned parameters, it can in principle extend to positions longer than those seen in training, though quality often degrades.
Main approaches
| Approach | How it works | Used in |
|---|---|---|
| Sinusoidal (absolute) | Fixed wave patterns added to embeddings | Original transformer (2017) |
| Learned absolute | A trainable vector per position | BERT, GPT-2 |
| Relative | Encodes the distance between tokens in attention scores | Transformer-XL, T5 |
| RoPE | Rotates query and key vectors by an angle based on position | Llama, Qwen, many modern LLMs |
| ALiBi | Adds a distance-based penalty to attention scores | Some long-context models |
| NoPE | No explicit encoding; causal models can infer some order | Research setting |
A worked example
Consider “Ravi paid Meera” and “Meera paid Ravi”. The token embeddings are identical sets. Without position information, attention would compute the same relationships and the model could not tell who paid whom. With positional encoding, the representation of “Ravi” at position 1 differs from “Ravi” at position 3, so the model can learn that the earlier name is the payer.
Absolute vs relative position
Absolute methods say “this is token 7”. Relative methods say “this token is three places before that one”. Relative approaches often generalise better to longer inputs, because language patterns usually depend on distance rather than exact index.
Positional methods and long context
Extending a model beyond its training length is mainly a positional problem. Techniques such as RoPE scaling, including NTK-aware scaling and YaRN, stretch positional patterns so a model trained on shorter sequences can handle longer ones after light fine-tuning. Choice of positional method is one reason models differ in how well they use long context.
Limitations
- Performance usually degrades beyond the trained length.
- Learned absolute encodings cannot extend past their table size.
- Position handling interacts with attention in ways that are hard to predict.
- Models often use position unevenly, attending more to the start and end of long inputs.