Batch size is the number of training examples processed together before the model updates its weights. Instead of learning from one example at a time, models process a group, average the gradients and take a single update step. Batch size affects memory use, training speed, stability and final quality, making it one of the most important training settings.
Key takeaways
- Larger batches give smoother, more reliable gradient estimates.
- Larger batches use more GPU memory.
- Batch size interacts strongly with the learning rate.
- In LLM training, batch size is usually measured in tokens, not examples.
- Gradient accumulation simulates large batches on small hardware.
Batch size explained in simple terms
Imagine a teacher marking homework and adjusting their teaching. They could change their lesson after each student’s paper, which is noisy and erratic. Or they could mark 30 papers, see the overall pattern and then adjust, which is steadier but slower to react. Batch size is how many papers are marked before adjusting.
The main trade-offs
| Aspect | Small batch | Large batch |
|---|---|---|
| Gradient noise | High | Low |
| Memory use | Low | High |
| Steps per epoch | Many | Few |
| GPU efficiency | Often poor | Usually better |
| Generalisation | Sometimes better, noise acts as regularization | Can need tuning to match |
| Stability | Noisier | Smoother |
A worked example
A fine-tuning dataset has 8,000 examples.
- Batch size 8: 1,000 steps per epoch, low memory, noisy updates.
- Batch size 64: 125 steps per epoch, more memory, smoother updates, usually faster wall-clock time if the GPU can hold it.
If memory only allows batch size 8 but you want 64, set gradient accumulation to 8: gradients from eight batches are summed and applied as one update. The effective batch size is 8 × 8 = 64.
Across multiple GPUs, effective batch size = per-device batch × accumulation steps × number of devices.
Batch size in LLM training
Large-scale pretraining usually reports batch size in tokens, for example “4 million tokens per batch”, because sequences vary in length. Very large batches are used to keep thousands of GPUs busy, combined with careful learning rate warmup and scaling.
Batch size and learning rate
They must be tuned together. When batch size increases, gradients become less noisy, so a larger learning rate is usually appropriate. A common heuristic is to scale the learning rate linearly with batch size, up to a limit, alongside a warmup period. Ignoring this is a frequent cause of training that diverges or stalls.
Batch size at inference
At inference time, batching means serving several user requests together. Larger inference batches raise throughput and lower cost per request, but can increase latency for individual users. Continuous batching balances the two by adding and removing requests dynamically.
Limitations
- Limited by available GPU memory.
- Very large batches can reduce generalisation without careful tuning.
- Optimal values differ by model, data and hardware.
- Changing batch size changes effective training dynamics, so results may not reproduce.