A hyperparameter is a configuration value chosen before training that controls how a model learns or how complex it is allowed to become. It is not learned from data, unlike weights and biases. The learning rate, batch size, number of layers, tree depth and regularisation strength are all hyperparameters.
Key takeaways
- Parameters are learned from data; hyperparameters are set by you.
- They control model capacity, the optimisation process and regularisation.
- Good defaults exist for most models and are a reasonable starting point.
- A few hyperparameters usually dominate results; most matter little.
- They are chosen using a validation set, never the test set.
Hyperparameters explained in simple terms
Baking bread, the recipe’s learned part is how the dough responds; the baker’s settings are oven temperature, baking time and tin size. Those settings are not derived from the dough: the baker chooses them in advance, and they determine whether the result is raw, perfect or burnt. Hyperparameters are the oven settings of machine learning.
Parameters vs hyperparameters
| Aspect | Parameters | Hyperparameters |
|---|---|---|
| Source | Learned during training | Set before training |
| Examples | Weights, biases | Learning rate, depth, batch size |
| Count | Thousands to billions | Typically fewer than 20 |
| Changed by | Gradient descent | Human choice or search |
| Selected using | Training data | Validation data |
Common hyperparameters by model type
| Model family | Key hyperparameters |
|---|---|
| Neural networks | Learning rate, batch size, epochs, optimiser, layers, hidden size, dropout, weight decay, warmup |
| Gradient boosting | Learning rate, number of trees, max depth, min samples per leaf, subsample, column sample, L1/L2 penalties |
| Random forest | Number of trees, max depth, max features, min samples split |
| Linear models | Regularisation type (L1 or L2), penalty strength |
| SVM | Kernel, C, gamma |
| k-means | Number of clusters, initialisation, runs |
| LoRA fine-tuning | Rank, alpha, target modules, learning rate, epochs |
| RAG systems | Chunk size, overlap, top-k, reranker depth |
Note the last row: pipeline settings are hyperparameters too, even when no model is being trained.
Which hyperparameters matter most
Effort is rarely evenly distributed. For most problems, a small number dominate:
- Learning rate for anything trained with gradient descent, by a wide margin.
- Regularisation strength, including weight decay, dropout and tree constraints.
- Model capacity: depth, width, number of trees.
- Batch size, which interacts with the learning rate.
- Training length, usually handled by early stopping.
Everything else is usually worth leaving at sensible defaults until these are settled.
A worked example
Fine-tuning a model with LoRA, a team records:
| Run | Learning rate | Rank | Epochs | Validation loss |
|---|---|---|---|---|
| 1 | 2e-5 | 8 | 3 | 0.84 |
| 2 | 2e-4 | 8 | 3 | 0.61 |
| 3 | 2e-4 | 32 | 3 | 0.63 |
| 4 | 2e-4 | 8 | 6 | 0.69 |
| 5 | 1e-3 | 8 | 3 | 1.42 |
The learning rate changed results dramatically; rank barely mattered; more epochs hurt through overfitting; a too-high rate destabilised training. This pattern, one dominant setting and several minor ones, is typical.
Finding good values
- Start from published defaults for your model and task.
- Change one thing at a time at first, to build intuition.
- Then use a systematic search; see hyperparameter tuning.
- Always select on validation data, and confirm once on the test set.
- Record every run’s settings and results, or results become irreproducible.
Limitations
- Optimal values are dataset-specific and do not transfer reliably.
- They interact, so tuning one at a time can miss good combinations.
- Searching costs compute, sometimes more than training the final model.
- Heavy tuning on a validation set can overfit to it.