Gradient clipping limits how large gradients can become before they are used to update a model’s weights. If the gradients in a step exceed a threshold, they are scaled down. This prevents exploding gradients, where one unusual batch produces an enormous update that destabilises or destroys a training run. It is standard practice in LLM training.
Key takeaways
- Clipping by global norm is the most common method, with a threshold often set to 1.0.
- It caps the size of updates without changing their direction.
- It protects long, expensive training runs from single bad batches.
- It addresses exploding gradients, not vanishing gradients.
- Frequent clipping can signal other problems, such as too high a learning rate.
Gradient clipping explained in simple terms
Think of a speed limiter on a truck. It does not change the direction of travel, only the maximum speed. Without it, one steep downhill stretch could send the truck out of control. Gradient clipping is the limiter: no matter how extreme one batch is, the update stays within a safe size.
Clipping by norm vs by value
| Method | What it does | Effect |
|---|---|---|
| Clip by global norm | Computes the overall gradient magnitude across all parameters and scales everything down if it exceeds the threshold | Preserves direction; standard for LLMs |
| Clip by value | Caps each individual gradient element at a maximum | Simpler, but can distort the update direction |
A worked example
Suppose the clipping threshold is 1.0 and a step produces a global gradient norm of 4.0.
- Scaling factor: 1.0 ÷ 4.0 = 0.25.
- Every gradient is multiplied by 0.25, giving a new norm of exactly 1.0.
- The direction of the update is unchanged; only its size is reduced.
If another step has a norm of 0.6, it is below the threshold and passes through untouched.
Why exploding gradients happen
- Unusual or corrupted training examples.
- Very deep networks multiplying signals across many layers.
- Recurrent models multiplying through many time steps.
- Learning rates that are too high.
- Numerical issues in low-precision training.
In large LLM runs, a single loss spike can undo hours of progress, so teams combine clipping with warmup, careful initialisation and monitoring, and sometimes restart from an earlier checkpoint when spikes occur.
Typical settings
- Threshold 1.0: the common default for transformer training.
- 0.5: used when training is unstable.
- 5.0 or higher: a loose safety net that rarely triggers.
Monitoring the gradient norm over time is useful. If clipping happens on almost every step, the learning rate is probably too high or the data contains problems. If it never happens, the threshold may be too loose to protect anything.
Where it fits in training
Clipping is applied after backpropagation computes gradients and after gradient accumulation averages them, but before the optimiser step. In mixed-precision training, gradients are unscaled first so the threshold applies to true values.
Limitations
- Does not help with vanishing gradients.
- Can mask deeper problems with data or hyperparameters.
- Too aggressive clipping slows learning.
- Adds a small computational cost each step.