The bias-variance tradeoff describes how a model’s prediction error splits into error from overly simple assumptions (bias) and error from excessive sensitivity to the specific training data (variance), and why reducing one usually increases the other. It is the framework that explains underfitting, overfitting and why there is an optimal level of model complexity.
Key takeaways
- Expected error = bias² + variance + irreducible noise.
- High bias means underfitting; high variance means overfitting.
- Increasing complexity usually lowers bias and raises variance.
- The best model minimises total error, not either component alone.
- Modern very large networks can break the classic curve, a phenomenon called double descent.
The tradeoff explained in simple terms
Think of archers. One shoots a tight cluster consistently left of the bullseye: that is high bias, low variance. Another scatters shots all around the target, averaging near centre: low bias, high variance. A third clusters tightly on target: low bias, low variance, which is the goal. Most model choices move you between the first two.
The error decomposition
| Component | Meaning | Reducible? |
|---|---|---|
| Bias² | Error from wrong or oversimplified assumptions | Yes, with a more flexible model or better features |
| Variance | Error from sensitivity to which data you happened to train on | Yes, with more data, regularisation or ensembles |
| Irreducible error | Noise inherent in the problem | No |
This decomposition was formalised for neural networks in 1992 research and holds for squared-error prediction.
How complexity changes the components
| Model complexity | Bias | Variance | Typical outcome |
|---|---|---|---|
| Very low (e.g. predicting the mean) | Very high | Very low | Severe underfitting |
| Low (linear model on curved data) | High | Low | Underfitting |
| Moderate | Moderate | Moderate | Usually best total error |
| High (deep unpruned tree) | Low | High | Overfitting |
| Very high with no regularisation | Very low | Very high | Memorisation |
Diagnosing which problem you have
| Observation | Diagnosis | Action |
|---|---|---|
| Training error high, validation error similar | High bias | More complex model, better features, train longer, less regularization |
| Training error low, validation error much higher | High variance | More data, regularisation, simpler model, early stopping |
| Both low and close | Good fit | Ship it, keep monitoring |
| Validation error varies a lot across cross-validation folds | High variance | Ensembling, more data |
Learning curves make this visible: plot training and validation error against training set size. If the curves converge at a high error, you have a bias problem and more data will not help. If a large gap persists, you have a variance problem and more data probably will.
A worked example
Predicting house prices with three models, measured on held-out data:
| Model | Training error | Validation error | Interpretation |
|---|---|---|---|
| Mean price only | 48 | 49 | Extreme bias |
| Linear regression | 22 | 24 | Still biased, but usable |
| Gradient boosting, tuned | 9 | 13 | Good balance |
| Deep tree, unpruned | 1 | 21 | Severe variance |
The unpruned tree has the lowest training error and the second-worst validation error. Training error alone would have chosen the worst model.
Remedies by component
Reduce bias: more expressive models, better feature engineering, longer training, weaker regularisation, remove restrictive assumptions.
Reduce variance: more training data, L1 or L2 regularization, dropout, bagging and random forests, early stopping, feature selection, data augmentation.
Ensembles are popular precisely because averaging many high-variance models cuts variance without adding much bias.
Double descent
Classical theory predicts that error rises once a model becomes complex enough to fit the training data perfectly. Research published in 2019 documented double descent: as models grow far beyond that point, test error can fall again. This helps explain why enormous neural networks generalise despite having more parameters than data points. The classic tradeoff remains a useful diagnostic tool, but it does not fully describe very large models.
Limitations
- The clean decomposition applies strictly to squared-error loss.
- Bias and variance cannot be measured directly without resampling.
- Does not describe modern overparameterised networks completely.
- Says nothing about fairness; see AI bias for that distinct meaning.