Hyperparameter tuning is the process of systematically searching for the hyperparameter values that make a model perform best on validation data. Rather than guessing, you define a search space, evaluate candidate configurations and select the best. Done well it produces real gains; done carelessly it wastes compute and quietly overfits the validation set.
Key takeaways
- Random search usually beats grid search for the same budget.
- Bayesian optimisation learns from past trials to choose the next ones.
- Successive halving methods kill weak runs early to save compute.
- Always tune on validation data, ideally with cross-validation.
- Tune the few settings that matter rather than everything at once.
Tuning explained in simple terms
Finding the best oven temperature and time for a recipe. You could try every combination in a grid, which is thorough and slow. You could try scattered combinations at random, which covers more distinct temperatures for the same number of bakes. Or you could learn from each attempt and try promising areas next, which is what Bayesian methods do.
The main search methods
| Method | How it works | Pros | Cons |
|---|---|---|---|
| Manual | Change settings by hand | Builds intuition | Slow, inconsistent |
| Grid search | Try every combination in a defined grid | Exhaustive, simple | Cost explodes with dimensions |
| Random search | Sample combinations randomly | More distinct values per setting; strong baseline | No learning between trials |
| Bayesian optimisation | Model the objective, pick promising points | Efficient on expensive runs | More complex; sequential |
| Hyperband / ASHA | Run many configs briefly, keep the best | Very compute-efficient | Needs partial results to be informative |
| Population-based training | Evolve a population during training | Tunes schedules over time | Complex to operate |
Research published in 2012 showed that random search finds good configurations more efficiently than grid search, because in a grid most settings are tested at only a few distinct values while the important ones need many.
A worked example: grid vs random
Suppose you tune learning rate and dropout with a budget of 25 runs.
- Grid search, 5 × 5: tests only 5 distinct learning rates.
- Random search, 25 samples: tests up to 25 distinct learning rates.
If learning rate dominates results and dropout barely matters, random search explores the important dimension five times more finely for the same cost. This is the core argument for random search as a default.
A practical workflow
- Baseline: train once with published defaults and record the score.
- Choose what to tune: typically learning rate, regularisation strength and capacity.
- Define ranges sensibly: sample learning rates on a log scale, for example 1e-5 to 1e-2.
- Pick a budget: decide how many runs you can afford before starting.
- Search: random search for 20–50 trials, or Bayesian for expensive runs.
- Refine: narrow ranges around the best region and search again.
- Validate: confirm the chosen configuration once on the test set.
- Record everything: settings, seeds, scores and data versions.
Avoiding common mistakes
| Mistake | Consequence | Fix |
|---|---|---|
| Tuning on the test set | Optimistic, meaningless results | Tune on validation only |
| Linear ranges for learning rates | Wastes trials in the wrong region | Sample log-uniformly |
| Tuning every setting at once | Budget spread thin | Focus on the dominant few |
| Ignoring seed variance | Chasing noise | Repeat best runs with different seeds |
| Over-searching a small validation set | Overfitting the validation set | Use cross-validation or a larger split |
| No logging | Irreproducible results | Track experiments systematically |
Nested cross-validation
When data is limited and you need an unbiased performance estimate and tuning, use nested cross-validation: an inner loop selects hyperparameters, an outer loop estimates performance. It is expensive, roughly multiplying the number of fits, but it avoids the optimistic bias of tuning and evaluating on the same splits.
Tuning for LLM fine-tuning
For fine-tuning, the search space is usually small: learning rate, epochs, LoRA rank and effective batch size. Since each run is expensive, teams typically test three learning rates on a subset, pick the best, and then tune only epochs via early stopping.
Limitations
- Compute cost can exceed the value of the improvement.
- Best settings do not transfer across datasets.
- Results can be noise, especially with small validation sets.
- Automated tuning can hide a poorly specified problem.