LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a model’s original weights and learns the update as the product of two small matrices. Because these matrices are tiny compared with the full weight matrices, LoRA trains a small fraction of the parameters, uses much less GPU memory and produces small adapter files. It was introduced by Edward Hu and colleagues at Microsoft in 2021 and is now the most widely used PEFT method for LLMs and image models.
Key takeaways
- LoRA keeps the base model frozen and trains small low-rank matrices.
- The rank (r) controls adapter size and capacity; common values range from 4 to 64.
- Adapters can be merged into the base weights, so there is no extra inference delay.
- QLoRA combines LoRA with a 4-bit base model to fine-tune very large models on one GPU.
- LoRA is also popular for customising image generators with new styles or subjects.
LoRA explained in simple terms
Imagine a huge spreadsheet of numbers, the model’s weights. Fine-tuning normally edits every cell. LoRA’s insight is that the changes needed for a new task usually follow a simple pattern, so you can describe them with two thin strips of numbers that, when multiplied, recreate the change across the whole sheet. You store and train only the strips.
How LoRA works
For a weight matrix W in the model, LoRA adds an update ΔW = B × A, where:
- A is a small matrix of size r × d (d is the layer width).
- B is a small matrix of size d × r.
- r is the rank, much smaller than d.
During training, W stays frozen; only A and B are updated. B usually starts at zero, so the model initially behaves exactly like the base model. A scaling factor, alpha, controls how strongly the update is applied. LoRA is typically applied to the attention projection matrices of transformer layers, and often to feed-forward layers too.
A worked example: counting parameters
Take one weight matrix of size 4,096 × 4,096, a common size in 7-billion-parameter models.
- Full fine-tuning: 4,096 × 4,096 = 16,777,216 trainable parameters.
- LoRA with r = 8: A is 8 × 4,096 = 32,768 and B is 4,096 × 8 = 32,768, for a total of 65,536.
That is about 0.4% of the original, for this matrix. Across a whole model, LoRA commonly trains well under 1% of the parameters. The original LoRA paper reported that, for GPT-3 175B, it cut trainable parameters by about 10,000 times and GPU memory by about 3 times compared with full fine-tuning, while matching quality on the tested tasks.
Choosing LoRA settings
| Setting | What it controls | Typical guidance |
|---|---|---|
| Rank (r) | Capacity of the adapter | Start at 8–16; increase for complex tasks |
| Alpha | Strength of the update | Often set equal to or double the rank |
| Target modules | Which layers get adapters | Attention layers; adding feed-forward layers can help |
| Learning rate | Speed of training | Usually higher than for full fine-tuning |
| Dropout | Regularization for the adapter | Small values, such as 0.05–0.1 |
QLoRA
QLoRA, introduced by Tim Dettmers and colleagues in 2023, loads the base model in 4-bit precision using quantization and trains LoRA adapters on top. The paper showed a 65-billion-parameter model could be fine-tuned on a single 48 GB GPU, putting large-model fine-tuning within reach of small teams.
LoRA in image generation
In the image community, a “LoRA” often means a small add-on file that teaches a diffusion model a new style, character or product. Designers can combine several LoRAs and adjust their strength, without retraining the full model.
Deploying LoRA adapters
- Merge the adapter into the base weights for a standalone model with no added latency.
- Keep separate to swap adapters per customer or task on one base model.
- Serving frameworks can host many LoRA adapters on a single GPU.
Limitations
- May trail full fine-tuning on large domain shifts.
- Rank and target modules need experimentation.
- Adapters only work with the exact base model they were trained on.
- Does not reduce the base model’s memory needs at inference unless combined with quantization.