Early stopping is a regularisation technique that halts training when validation performance stops improving. Training loss almost always keeps falling, but validation loss eventually turns upward as the model begins memorising rather than generalising. Early stopping detects that turning point and stops, usually restoring the best weights seen so far.
Key takeaways
- It monitors a validation metric, never the training metric.
- Patience is how many epochs of no improvement to tolerate before stopping.
- Best practice is to restore the best checkpoint, not keep the last one.
- It prevents overfitting and saves compute at the same time.
- It is effectively free, which is why it is almost always worth enabling.
Early stopping explained in simple terms
A student revising for an exam improves steadily, then starts memorising past papers word for word and performs worse on new questions. Early stopping is noticing the moment extra revision stops helping, and stopping there. The difficulty is that noise means a single bad week does not prove the peak has passed, which is what patience handles.
How it works
- After each epoch, evaluate on the validation set.
- If the metric improved beyond a minimum threshold, save a checkpoint and reset the patience counter.
- If it did not improve, increment the counter.
- When the counter reaches the patience limit, stop training.
- Restore the best checkpoint.
The key settings
| Setting | Meaning | Typical value |
|---|---|---|
| Monitor | Metric to watch | Validation loss, or the business metric |
| Patience | Epochs without improvement before stopping | 3–10 for small models; fewer for expensive training |
| Min delta | Smallest change counted as improvement | A small value suited to the metric’s scale |
| Mode | Whether lower or higher is better | Min for loss, max for accuracy |
| Restore best weights | Roll back to the best epoch | Almost always on |
A worked example
| Epoch | Training loss | Validation loss | Patience counter |
|---|---|---|---|
| 1 | 0.68 | 0.64 | 0 |
| 2 | 0.51 | 0.48 | 0 |
| 3 | 0.42 | 0.44 | 0 (best) |
| 4 | 0.36 | 0.45 | 1 |
| 5 | 0.31 | 0.47 | 2 |
| 6 | 0.27 | 0.46 | 3 → stop |
Training loss falls throughout; validation loss bottoms out at epoch 3. With patience 3, training stops after epoch 6 and the epoch 3 weights are restored. Without early stopping, running to epoch 20 would have produced a worse model and burned compute.
Why patience matters
Validation curves are noisy. A single worse epoch may be random variation, especially with a small validation set or a high learning rate. Patience of 1 stops too early and leaves performance on the table. Very large patience wastes compute and risks drifting into a much worse region. Three to ten epochs is a reasonable range for most training, scaled down when each epoch is expensive.
Interaction with other settings
- Learning rate schedules: with cosine decay, the best results often come at the very end, so aggressive early stopping can cut training before the decay has done its work. Consider reduce-on-plateau schedules alongside early stopping.
- Regularisation: with strong dropout or weight decay, the validation curve often flattens rather than rising sharply, so min delta matters more.
- Gradient boosting: libraries implement early stopping rounds, which is standard practice for choosing the number of trees.
- LLM fine-tuning: runs are short, often 1–3 epochs, so evaluate several times per epoch rather than once.
Common pitfalls
| Pitfall | Consequence |
|---|---|
| Monitoring training loss | Never triggers; defeats the purpose |
| Using the test set to stop | Contaminates the final estimate |
| Not restoring best weights | Ships a worse model than you trained |
| Validation set too small | Noisy signal, premature or late stopping |
| Evaluating too rarely | Misses the true optimum |
Limitations
- Needs a held-out validation set, which costs training data.
- Noisy validation curves make the stopping point approximate.
- Interacts awkwardly with some learning rate schedules.
- Does not replace other regularisation; it complements it.