A training epoch is one complete pass of the training process through the entire training dataset. After one epoch, the model has seen every training example once. Most machine learning models train for several epochs; large language model pretraining, by contrast, often uses a single pass or less over an enormous corpus.
Key takeaways
- One epoch = every example seen once.
- An epoch is made of many steps, each processing one batch.
- Steps per epoch = dataset size ÷ batch size.
- LLM fine-tuning usually uses 1–3 epochs; more often causes overfitting.
- Pretraining usually makes roughly one pass over deduplicated data.
An epoch explained in simple terms
Think of revising a book of practice questions. Working through the whole book once is one epoch. Most students go through it two or three times, and each pass improves recall. But go through it too many times and you start memorising the answers rather than learning the subject. Models behave the same way.
Epochs, steps and batches
Worked example: a fine-tuning dataset of 10,000 examples with a batch size of 16.
- Steps per epoch: 10,000 ÷ 16 = 625 steps.
- Training for 3 epochs: 625 × 3 = 1,875 steps.
- With gradient accumulation of 4, the model updates its weights every 4 steps, so there are about 469 weight updates per epoch with an effective batch size of 64.
| Term | Meaning |
|---|---|
| Batch | A group of examples processed together |
| Step (iteration) | One batch processed, usually one weight update |
| Epoch | One full pass through the dataset |
How many epochs should you use?
| Scenario | Typical epochs |
|---|---|
| LLM pretraining | Around 1 pass over a large deduplicated corpus |
| Instruction fine-tuning | 1–3 |
| LoRA fine-tuning on small data | 2–5 |
| Small classical ML models | 10–100 or more |
The reliable method is not to fix a number in advance but to watch the validation loss. Train while it improves; stop when it flattens or rises. This is early stopping.
Signs you have trained too long
- Training loss keeps falling while validation loss rises.
- Outputs start reproducing training examples word for word.
- The model becomes worse at tasks outside the fine-tuning data, a sign of catastrophic forgetting.
Why LLM pretraining uses so few epochs
Pretraining corpora are so large that a single pass already involves trillions of tokens and enormous compute. Repeating data too often gives diminishing returns and can increase memorisation of specific text, which raises privacy and copyright concerns. Research has explored repeating high-quality data a few times when fresh data is limited, with careful monitoring.
Limitations
- The right number of epochs depends on data size, quality and model size.
- Epoch counts alone do not describe training; batch size and learning rate matter too.
- With streaming or continuously growing data, the idea of an epoch is less meaningful.