Gradient accumulation is a training technique that adds up gradients from several small batches and applies them as a single weight update. This simulates training with a large batch size while only ever holding one small batch in GPU memory. It is one of the most practical tricks for fine-tuning large models on limited hardware.
Key takeaways
- Effective batch size = per-device batch × accumulation steps × number of devices.
- It trades wall-clock time for memory: the same total work, fewer weight updates.
- It is mathematically close to true large-batch training, with minor differences.
- Gradients must be scaled correctly to avoid inflating the learning rate effect.
- Widely supported in training frameworks with a single setting.
Gradient accumulation explained in simple terms
You want to carry 64 bricks but can only lift 8 at a time. Instead of giving up, you make eight trips and stack the bricks in one pile before the wall is built. The wall is built once, from all 64 bricks. Gradient accumulation does the same with gradients: several small passes, one update.
How it works
- Process a small batch and compute gradients.
- Add them to a running total instead of updating weights.
- Repeat for N batches.
- Divide the total by N to get the average gradient.
- Apply one optimiser update and reset the accumulator.
A worked example
You want an effective batch size of 64, but your GPU only fits 8 examples.
| Setting | Value |
|---|---|
| Per-device batch size | 8 |
| Accumulation steps | 8 |
| GPUs | 1 |
| Effective batch size | 8 × 8 × 1 = 64 |
With 8,000 training examples, one epoch has 1,000 forward and backward passes but only 125 weight updates. Peak memory matches a batch of 8, while the learning behaviour resembles a batch of 64.
When to use it
- Fine-tuning large models on a single consumer or mid-range GPU.
- Training with long sequences, which consume memory quickly.
- Matching a published recipe’s effective batch size on different hardware.
- Stabilising training when small batches make loss noisy.
Common mistakes
| Mistake | Consequence |
|---|---|
| Not averaging accumulated gradients | Effective learning rate becomes too large |
| Forgetting to zero gradients after the update | Gradients keep piling up |
| Counting steps inconsistently | Schedulers and logging misalign |
| Ignoring batch normalisation layers | Statistics are computed per small batch, unlike true large batches |
Most modern training libraries handle the averaging and step counting automatically when accumulation is configured.
Gradient accumulation vs other memory techniques
| Technique | Saves memory by |
|---|---|
| Gradient accumulation | Processing fewer examples at a time |
| Gradient checkpointing | Recomputing activations instead of storing them |
| Mixed precision | Using 16-bit numbers for most operations |
| QLoRA | Quantizing the base model and training small adapters |
These combine well: QLoRA with gradient checkpointing and accumulation lets very large models be fine-tuned on a single GPU.
Limitations
- Training takes longer in wall-clock time, since updates are less frequent.
- Does not reduce memory for the model weights themselves.
- Batch-statistics layers behave slightly differently from true large batches.
- Very high accumulation counts can make training slow to converge in practice.