Prefix tuning is a parameter-efficient fine-tuning method that learns trainable vectors and inserts them into the attention keys and values at every layer of a frozen model. Where prompt tuning only adds vectors at the input, prefix tuning influences the model at every depth, which gives stronger control with still very few trainable parameters. It was introduced by Xiang Lisa Li and Percy Liang in 2021.
Key takeaways
- Prefixes act like extra virtual tokens that every attention layer can attend to.
- The base model’s weights stay frozen.
- It typically trains around 0.1–3% of model parameters.
- It usually outperforms input-only prompt tuning, especially on smaller models.
- LoRA has largely replaced it in practice for LLMs.
Prefix tuning explained in simple terms
Imagine briefing a team of reviewers. Prompt tuning hands a note to the first reviewer and hopes the message carries through the chain. Prefix tuning hands the same note to every reviewer at every stage, so the guidance is present throughout the process rather than fading as work moves along.
How it works
- Create trainable vectors for each layer, usually a prefix length of 10 to 100 virtual positions.
- Split them into key and value vectors for that layer’s self-attention.
- Prepend them to the real keys and values, so every real token can attend to the prefix.
- Train only these vectors; the frozen model is untouched.
- Store the prefix as the task-specific artefact.
Because directly optimising these vectors can be unstable, the original method trained them through a small network, a technique called reparameterisation, which is discarded after training.
Worked example: a model with 32 layers, dimension 4,096 and a prefix of 20 positions stores keys and values for each layer: 32 × 2 × 20 × 4,096 ≈ 5.2 million parameters. That is far more than input-only prompt tuning, but still under 0.1% of a 7-billion-parameter model.
Prefix tuning vs other PEFT methods
| Method | Where it acts | Typical trainable share | Inference cost |
|---|---|---|---|
| Prompt tuning | Input embeddings only | About 0.001% | Extra input tokens |
| Prefix tuning | Keys and values in every layer | About 0.1–3% | Longer effective attention sequence |
| Adapters | New layers in each block | 0.5–5% | Extra layer compute |
| LoRA | Low-rank updates to weight matrices | Under 1% | None once merged |
Where it is used
Prefix tuning showed strong results on generation tasks such as summarisation and table-to-text, and it remains useful in research on controllable generation. Some multimodal systems use prefix-style vectors to feed image features into a frozen language model.
Limitations
- Prefix positions consume part of the attention sequence at every layer.
- Training can be sensitive without reparameterisation.
- Usually trails LoRA on modern LLM tasks.
- Prefixes are tied to a specific base model and cannot be merged into weights.