Model variance is the amount a machine learning model’s predictions would change if it were trained on a different sample of data from the same source. A high-variance model is overly sensitive to the specific examples it saw, so it captures noise and overfits. A low-variance model gives similar results regardless of which sample it was trained on.
Key takeaways
- High variance is the main cause of overfitting.
- Complex, flexible models tend to have higher variance.
- Variance trades off against bias.
- More data, regularisation and ensembles are the main ways to reduce variance.
Model variance explained in simple terms
Picture an archer whose arrows land all over the target: sometimes top left, sometimes bottom right. On average they may centre on the bullseye, but each shot is unpredictable. That scatter is high variance. A high-variance model behaves the same way. Train it on one sample and it predicts one thing; train it on a slightly different sample and it predicts something quite different.
How to recognise high variance
| Signal | Meaning |
|---|---|
| Large gap between training and validation scores | Model fits training data far better than new data |
| Cross-validation scores vary widely between folds | Results depend heavily on which data was used |
| Retraining on new samples changes predictions a lot | Model is unstable |
| Validation score improves when adding data | Model needed more examples to stabilise |
What causes high variance?
- High model complexity: deep decision trees, high-degree polynomials, large neural networks.
- Small datasets: fewer examples means each one has more influence.
- Many features: especially noisy or irrelevant ones.
- Noisy labels: the model learns random errors as patterns.
- Weak regularisation: nothing stops the model fitting every detail.
How to reduce model variance
| Technique | Why it works |
|---|---|
| More training data | Noise averages out and true patterns dominate |
| Regularisation (L1, L2, dropout) | Discourages overly complex solutions |
| Simpler models | Fewer parameters to fit noise |
| Bagging and random forests | Averaging many models trained on different samples cancels out individual errors |
| Feature selection | Fewer noisy inputs |
| Early stopping | Stops training before the model memorises data |
| Data augmentation | Adds realistic variety to training examples |
The bias-variance trade-off
Total prediction error combines bias squared, variance and irreducible noise. Simple models have high bias and low variance; complex models have low bias and high variance. The aim is the model complexity where total error on the validation set is lowest. Ensembles are popular because they reduce variance without adding much bias: random forests average many high-variance decision trees into a stable model.
Variance in deep learning
Very large neural networks can have huge capacity yet still generalise well when trained on large datasets with regularisation. Researchers have documented “double descent”, where error can fall again as models grow past the point where they fit training data perfectly. The classic trade-off remains a useful guide, but it does not describe every modern model.
Model variance vs data variance
Do not confuse model variance with the statistical variance of a dataset, which measures how spread out values are. Model variance is about how unstable a model’s predictions are across different training samples.
Why it matters
A high-variance model can look excellent in development and behave unpredictably in production. It may also change noticeably every time it is retrained, which confuses users and makes auditing harder. Stable models build trust.