Adapter tuning is a parameter-efficient fine-tuning method that inserts small trainable layers, called adapters, inside a frozen pretrained model. Only the adapters are trained, so a new task needs a few million parameters rather than billions. Introduced by Neil Houlsby and colleagues in 2019, adapter tuning was one of the first practical alternatives to full fine-tuning and paved the way for methods such as LoRA.
Key takeaways
- Adapters are small modules added between or inside existing layers.
- The base model’s weights stay frozen.
- Typical bottleneck adapters train roughly 0.5–5% of the model’s parameters.
- Adapters can be swapped per task, but they add a little inference latency.
- LoRA has largely overtaken classic adapters for LLMs, partly because it can be merged away.
Adapter tuning explained in simple terms
Think of a factory production line you are not allowed to rebuild. You can, however, add small adjustment stations between existing machines. Each station tweaks the product for a particular order. Swap the stations and the same line produces something different. Adapters are those small stations inside a frozen model.
How a bottleneck adapter works
A standard adapter is a tiny two-layer network inserted into each transformer block:
- Down-projection: reduce the representation from the model dimension, say 4,096, to a small bottleneck size, say 64.
- Non-linearity: apply an activation function.
- Up-projection: expand back to 4,096.
- Residual connection: add the result to the original input, so an untrained adapter starts close to a no-op.
Worked example: with a model dimension of 4,096 and a bottleneck of 64, one adapter has about 4,096 × 64 × 2 ≈ 524,000 parameters. Across 32 layers with two adapters each, that is roughly 33 million parameters, a small fraction of a 7-billion-parameter model.
Adapter variants
| Variant | Idea |
|---|---|
| Bottleneck (Houlsby) | Two adapters per transformer block |
| Pfeiffer adapters | One adapter per block, often as effective and faster |
| Parallel adapters | Run alongside a layer instead of after it |
| AdapterFusion | Combine several task adapters at inference |
| Language and task adapters | Stack a language adapter with a task adapter |
Adapters vs LoRA
| Aspect | Bottleneck adapters | LoRA |
|---|---|---|
| Where added | New layers inside the block | Low-rank updates to existing weight matrices |
| Inference latency | Small increase; extra layers must run | None once merged into base weights |
| Composability | Strong, with fusion methods | Multiple adapters can be combined, with care |
| Popularity for LLMs | Declining | Dominant |
| Modularity | Excellent for multi-task setups | Excellent for per-customer models |
The key practical difference is merging: LoRA’s update can be folded into the base weights, so serving costs nothing extra. Adapter layers cannot be merged away, which matters at scale.
Where adapters are still useful
- Multi-task and multilingual setups where modules are combined deliberately.
- Research on modular and composable models.
- Vision and speech models, where adapter-style modules remain common.
- Settings where clear separation between base and task modules is preferred.
Limitations
- Adds inference latency, unlike merged LoRA.
- Changes the model’s architecture, so serving stacks must support it.
- May slightly trail full fine-tuning on large domain shifts.
- Adapters are tied to the exact base model they were trained on.