Backpropagation (short for “backward propagation of errors”) is the algorithm that calculates how much each weight in a neural network contributed to the prediction error. It starts at the output, where the error is measured, and works backwards layer by layer using the chain rule of calculus. The results, called gradients, tell gradient descent how to adjust each weight to reduce the error.
Key takeaways
- Backpropagation computes gradients; gradient descent uses them to update weights.
- It relies on the chain rule to pass error signals backwards through layers.
- It is efficient: one backward pass gives gradients for every weight at once.
- It was popularised for neural networks in 1986 by David Rumelhart, Geoffrey Hinton and Ronald Williams.
- Almost every modern deep learning model is trained with it.
Backpropagation explained in simple terms
Imagine a relay team loses a race by three seconds. The coach watches the replay backwards: the last runner lost one second, but partly because the third runner handed over the baton late, which was partly because the second runner started slowly. By tracing backwards, the coach works out how much of the lost time each runner caused, and gives each one specific advice. Backpropagation does this for every weight in a network.
The learning loop
- Forward pass: input flows through the network to produce a prediction.
- Loss: a loss function measures how wrong the prediction is.
- Backward pass (backpropagation): the error signal flows back through each layer, computing the gradient of the loss for every weight and bias.
- Update: an optimiser such as gradient descent or Adam adjusts each parameter slightly against its gradient.
- Repeat across many batches and epochs.
The chain rule, briefly
A network is a chain of functions: layer 3 depends on layer 2, which depends on layer 1. The chain rule says the effect of an early weight on the final loss equals the product of the effects along the path: how the loss changes with the output, times how the output changes with the hidden layer, times how the hidden layer changes with that weight. Backpropagation computes these products efficiently by reusing results from later layers as it moves backwards.
A worked example: one weight
Take the simplest possible network: prediction = w × x. Let x = 2, weight w = 0.5, and the true answer y = 3. Use squared error loss = (prediction − y)².
- Forward pass: prediction = 0.5 × 2 = 1. Loss = (1 − 3)² = 4.
- Backward pass: how does the loss change with w? The chain rule gives: d(loss)/d(prediction) = 2 × (1 − 3) = −4, and d(prediction)/d(w) = x = 2. So the gradient = −4 × 2 = −8.
- Update with learning rate 0.1: new w = 0.5 − 0.1 × (−8) = 1.3.
- Check: prediction = 1.3 × 2 = 2.6, loss = (2.6 − 3)² = 0.16. The error fell from 4 to 0.16 in one step.
Real networks do exactly this, but for millions or billions of weights at once, using matrix maths on GPUs.
A brief history
The mathematics behind backpropagation, reverse-mode automatic differentiation, was described by Seppo Linnainmaa in 1970. Paul Werbos proposed applying it to neural networks in his 1974 thesis. The 1986 paper by Rumelhart, Hinton and Williams showed that it could train multi-layer networks to learn useful internal representations, reviving interest in neural networks. Today, frameworks such as PyTorch and TensorFlow perform backpropagation automatically.
Common problems
| Problem | What happens | Common fixes |
|---|---|---|
| Vanishing gradients | Gradients shrink through many layers, so early layers barely learn | ReLU activations, residual connections, careful initialisation, normalisation |
| Exploding gradients | Gradients grow huge and training becomes unstable | Gradient clipping, lower learning rate, normalisation |
| Memory use | Activations from the forward pass must be stored | Smaller batches, gradient checkpointing |
| Recurrent networks | Gradients must flow back through time | Backpropagation through time, LSTM and GRU gates |
Backpropagation vs gradient descent
These are often confused. Backpropagation answers “which direction should each weight move, and how strongly?”. Gradient descent answers “take the step”. You need both to train a network.
Limitations
- Requires the whole network to be differentiable.
- Stores intermediate values, which uses a lot of memory for large models.
- Can suffer from vanishing and exploding gradients.
- Is not how the brain is believed to learn, which motivates research into alternatives.