Stochastic gradient descent (SGD) is a version of gradient descent that updates model weights using one example or a small batch at a time, rather than computing gradients over the entire dataset. “Stochastic” means random: each step uses a random sample, so the direction is noisy but far cheaper to compute. Almost all modern deep learning training is some form of SGD.
Key takeaways
- Batch gradient descent uses all data per update; SGD uses a sample.
- In practice, “SGD” almost always means mini-batch SGD.
- The noise is not purely a drawback: it helps escape saddle points and flat regions.
- Momentum and adaptive methods such as Adam build directly on SGD.
- SGD with momentum is still competitive, and sometimes generalises better than Adam.
SGD explained in simple terms
Imagine finding the lowest point in a valley in fog. Batch gradient descent surveys the entire landscape before each careful step: accurate but painfully slow. SGD glances at a small patch of ground and steps immediately. Each step is less reliable, but you take thousands in the time the careful surveyor takes one, and you arrive sooner.
The three variants
| Variant | Data per update | Updates per epoch (10,000 examples) | Character |
|---|---|---|---|
| Batch | All 10,000 | 1 | Stable, slow, memory-heavy |
| Stochastic (pure) | 1 | 10,000 | Very noisy, poor hardware use |
| Mini-batch | e.g. 32 | 313 | The practical standard |
Pure single-example SGD is rare today because GPUs are far more efficient on batches. When people say SGD, they nearly always mean mini-batch SGD with a batch size between 16 and several thousand.
A worked example
A model has weight w = 0.80 and a learning rate of 0.1.
| Step | Mini-batch gradient | Update | New w |
|---|---|---|---|
| 1 | 0.5 | 0.80 − 0.1 × 0.5 | 0.75 |
| 2 | −0.2 | 0.75 − 0.1 × (−0.2) | 0.77 |
| 3 | 0.6 | 0.77 − 0.1 × 0.6 | 0.71 |
The gradients disagree between batches because each sees different examples. The path wobbles, but across thousands of steps the average direction is correct and the weight converges towards a good value.
Why noise helps
- It can knock the model out of saddle points and flat regions where exact gradients stall.
- It acts as a mild regulariser, discouraging very sharp minima.
- Research has linked the noise scale, driven by batch size and learning rate, to how well models generalise.
Too much noise, from a tiny batch or a large learning rate, prevents convergence. Too little, from a very large batch, can lead to sharper solutions that generalise less well unless the learning rate is scaled and warmed up.
Improvements built on SGD
| Method | Idea | Effect |
|---|---|---|
| Momentum (1964) | Accumulate a velocity across steps | Smooths zig-zags, speeds up along consistent directions |
| Nesterov momentum | Look ahead before computing the gradient | Slightly faster convergence |
| AdaGrad | Per-parameter step sizes based on past gradients | Helps sparse features |
| RMSProp | Moving average of squared gradients | Stabilises step sizes |
| Adam (2014) | Momentum plus adaptive step sizes | The common default for deep learning |
| AdamW | Adam with decoupled weight decay | Standard for training large models |
SGD vs Adam in practice
| Consideration | SGD with momentum | Adam / AdamW |
|---|---|---|
| Tuning effort | Higher; sensitive to learning rate | Lower; works near defaults |
| Memory | Lower | Stores two extra values per parameter |
| Convergence speed | Slower early | Faster early |
| Final generalisation | Sometimes better, notably in vision | Excellent for transformers and LLMs |
Most large language models are trained with AdamW. Many computer vision results still use SGD with momentum, which is why the choice is empirical rather than settled.
Practical settings
- Shuffle data every epoch so batches are genuinely random.
- Scale the learning rate with batch size, and use a warmup phase.
- Use a learning rate scheduler, typically cosine decay.
- Apply gradient clipping to survive outlier batches.
- Track training loss over a moving window, since per-batch loss is inherently noisy.
Limitations
- Noisy loss curves make progress harder to read.
- Sensitive to learning rate and batch size choices.
- Convergence is to a good solution, not a provably optimal one.
- Requires shuffling and careful data ordering to avoid bias.