Gradient descent is an optimisation algorithm that trains machine learning models by repeatedly adjusting their parameters in the direction that reduces error the fastest. It calculates how the loss function changes with each parameter (the gradient), then takes a small step “downhill”. Repeating this thousands or millions of times moves the model toward low error. It is the engine behind most neural network training.
Key takeaways
- Gradient descent minimises a loss function by following its slope downward.
- The learning rate controls step size and is one of the most important settings.
- Main variants are batch, stochastic and mini-batch gradient descent.
- Modern optimisers such as Adam build on gradient descent with adaptive step sizes.
Gradient descent explained in simple terms
Imagine standing on a foggy hillside at night, trying to reach the valley. You cannot see the bottom, but you can feel the slope under your feet. So you take a step in the steepest downhill direction, check the slope again, and step again. Eventually you reach low ground. The hill is the loss function, your position is the model’s parameters, and each step is one update.
How does gradient descent work?
- Start with initial parameter values, often random.
- Predict on training data and calculate the loss.
- Compute the gradient: the direction and steepness of the loss for each parameter. In neural networks, backpropagation calculates this efficiently.
- Update each parameter: new value = old value − learning rate × gradient.
- Repeat until the loss stops improving meaningfully.
The method dates back to the mathematician Augustin-Louis Cauchy, who described it in 1847. It became central to machine learning once combined with backpropagation for neural networks.
The learning rate
| Learning rate | Effect |
|---|---|
| Too small | Very slow progress; may look like underfitting |
| Well chosen | Steady, efficient convergence |
| Too large | Overshoots the minimum; loss jumps around or explodes |
Many teams use a learning rate schedule, starting larger and shrinking over time, or a short warm-up period at the start.
Types of gradient descent
| Type | Data per update | Pros | Cons |
|---|---|---|---|
| Batch | Entire dataset | Stable, accurate steps | Slow and memory-heavy for large data |
| Stochastic (SGD) | One example | Fast, noisy steps can escape poor areas | Very noisy updates |
| Mini-batch | Small batch, often 32–512 examples | Balance of speed and stability; suits GPUs | Batch size needs tuning |
In practice, “SGD” often refers to mini-batch gradient descent.
Popular optimisers
- Momentum: keeps moving in consistent directions, smoothing zig-zags.
- RMSProp: adapts step size for each parameter.
- Adam (introduced in 2014): combines momentum and adaptive step sizes; a common default for deep learning.
- AdamW: a variant of Adam with improved weight decay, widely used for training large models.
Common challenges
- Local minima: low points that are not the lowest overall.
- Saddle points and plateaus: flat regions where progress stalls.
- Vanishing and exploding gradients: gradients become tiny or huge in deep networks.
- Poor feature scaling: features on very different scales slow convergence.
In large neural networks, poor local minima turn out to be less of a problem than once feared; flat regions and saddle points are more common obstacles.
Gradient descent and overfitting
Gradient descent keeps reducing training loss as long as it runs. Left too long, the model can overfit. Early stopping uses the validation set to halt training when validation loss stops improving.
Limitations
Gradient descent needs a loss function that can be differentiated. It does not guarantee the global best solution for complex models. Results depend on the learning rate, initialisation and batch size. And training very large models with it is computationally expensive.