A learning rate scheduler is a rule that changes the learning rate during training rather than keeping it fixed. Almost all modern LLM training uses one: the rate starts near zero, rises during a warmup phase, then decays towards the end. This gives stability at the start, fast progress in the middle and fine adjustment at the end.
Key takeaways
- Schedulers usually combine warmup with a decay curve.
- Warmup prevents early instability when weights are random and gradients are large.
- Cosine decay is the most common shape in LLM training.
- Decay allows the model to settle into a good solution rather than bouncing around.
- Warmup is commonly a small fraction of total steps, often around 1–5%.
A scheduler explained in simple terms
Think of driving into an unfamiliar town. You start slowly while you get your bearings (warmup), drive at a good speed once the road is clear (peak learning rate), then slow down as you approach the address so you do not overshoot (decay). A fixed speed the whole way would be either dangerous at the start or wasteful in the middle.
Why warmup matters
At the beginning of training, weights are random, gradients can be large and erratic, and adaptive optimisers such as Adam have not yet built reliable statistics. Jumping straight to the full learning rate often causes loss spikes or divergence. Warmup raises the rate gradually, usually linearly, over the first steps.
Common schedules
| Schedule | Shape | Notes |
|---|---|---|
| Constant | Flat | Simple; rarely optimal for large runs |
| Linear decay | Straight decline to zero | Common for fine-tuning |
| Cosine decay | Smooth curve, slow at first, faster near the end | Standard for LLM pretraining |
| Cosine with warm restarts | Repeated cycles | Helps escape poor regions |
| Step decay | Drops at fixed milestones | Traditional in vision models |
| Inverse square root | Decays with the square root of steps | Used in the original transformer |
| Warmup-stable-decay | Long stable phase, then a short decay | Useful when total steps are not fixed in advance |
A worked example
A fine-tuning run has 2,000 total steps, a peak learning rate of 2e-5 and 5% warmup.
- Warmup: steps 1–100, the rate rises linearly from 0 to 2e-5.
- Decay: steps 101–2,000, cosine decay from 2e-5 down towards 0.
- At step 1,050, roughly halfway through decay, the rate is about 1e-5.
Ending near zero means the final updates are small, letting the model settle rather than jump.
Choosing a scheduler
- Fine-tuning: linear or cosine decay with short warmup works well.
- Pretraining: cosine decay with warmup is standard; some teams keep a small final rate rather than zero.
- Unknown training length: warmup-stable-decay lets you stop at any point and apply a short decay.
- LoRA: the same schedules apply, usually with a higher peak rate.
Interaction with other settings
Schedulers work with, not instead of, good choices of peak learning rate, batch size and optimiser. Changing total steps changes the whole curve, so restarting a run with different length requires care, especially when resuming from a checkpoint.
Limitations
- Adds hyperparameters: warmup length, final rate, curve shape.
- Cosine schedules assume a known total number of steps.
- Wrong schedules waste compute on expensive runs.
- Benefits vary; simple linear decay is often nearly as good for small jobs.