The learning rate is the setting that controls how large each weight update is during training. After backpropagation computes gradients, the optimiser multiplies them by the learning rate before adjusting the weights. It is widely regarded as the single most important hyperparameter: get it wrong and training either diverges or crawls.
Key takeaways
- Update = current weight − learning rate × gradient.
- Too high causes loss spikes or divergence; too low means slow or stalled learning.
- Typical values: around 1e-4 to 3e-4 for LLM pretraining, 1e-5 to 5e-5 for full fine-tuning, and 1e-4 to 3e-4 for LoRA.
- It is almost always varied over training using a scheduler.
- It must be tuned together with batch size.
A learning rate explained in simple terms
Picture walking downhill in fog towards the lowest point in a valley, as in gradient descent. The learning rate is your stride length. Tiny steps mean you will get there eventually, but very slowly. Huge steps may carry you straight over the valley and up the opposite slope, bouncing back and forth without settling.
A worked example
A weight is 0.50 and the gradient is 2.0, meaning increasing the weight would increase the loss.
| Learning rate | New weight | Effect |
|---|---|---|
| 0.001 | 0.498 | Tiny change, slow progress |
| 0.01 | 0.48 | Steady progress |
| 1.0 | −1.50 | Large overshoot, likely instability |
Across billions of parameters, these effects compound. A too-large learning rate can send the loss to infinity within a few steps.
Signs your learning rate is wrong
| Symptom | Likely cause |
|---|---|
| Loss spikes, becomes NaN or diverges | Learning rate too high |
| Loss falls extremely slowly | Learning rate too low |
| Loss oscillates without improving | Slightly too high |
| Loss drops then plateaus early | Needs decay, or a scheduler |
| Model forgets general abilities during fine-tuning | Too high for the task |
Typical values by scenario
| Scenario | Common range |
|---|---|
| LLM pretraining | 1e-4 to 3e-4 (with warmup and decay) |
| Full fine-tuning of an LLM | 1e-5 to 5e-5 |
| LoRA or other PEFT | 1e-4 to 3e-4 |
| Small classical models | 1e-3 to 1e-2 |
These are starting points, not rules. Always check the base model’s documentation, since recommended values differ.
Adaptive optimisers
Optimisers such as Adam and AdamW adapt the step size per parameter using running estimates of gradient statistics. This makes training far less sensitive to the exact learning rate, but does not remove the need to choose a sensible base value.
How to find a good learning rate
- Start from published values for a similar model and task.
- Run short trials at several values, such as 1e-5, 5e-5 and 1e-4.
- Compare training and validation loss curves.
- Choose the highest value that stays stable, then add warmup and decay.
- Retune if you change batch size significantly.
Limitations
- No universal best value.
- Interacts with batch size, optimiser, model size and data.
- A good rate early may be too high later, hence schedulers.
- Small changes can noticeably alter results, complicating reproducibility.