Cross-validation is a technique that repeatedly splits data into training and validation parts, rotating which part is held out, then averages the results. A single split can be lucky or unlucky, especially with limited data. Cross-validation uses every example for validation exactly once, giving a more stable estimate of how a model will perform on new data.
Key takeaways
- K-fold is the standard: split into k parts, train k times, average the scores.
- 5 or 10 folds are the usual choices.
- Stratified folds preserve class proportions and are the default for classification.
- Time series and grouped data need specialised splitting, or results are invalid.
- Cross-validation is for model selection and estimation, not a replacement for a final test set.
Cross-validation explained in simple terms
Testing a recipe on one friend tells you little: they may simply dislike coriander. Testing it on five friends in turn, each tasting a different batch, gives a far better picture. Cross-validation does this with data: every example gets a turn as the judge.
How k-fold cross-validation works
- Shuffle the data (unless it is time-ordered).
- Split it into k equal folds.
- For i = 1 to k: train on all folds except i, validate on fold i.
- Record the score each time.
- Report the mean and the standard deviation.
Worked example with 5-fold on 1,000 examples:
| Fold | Training examples | Validation examples | Accuracy |
|---|---|---|---|
| 1 | 800 | 200 | 0.86 |
| 2 | 800 | 200 | 0.83 |
| 3 | 800 | 200 | 0.88 |
| 4 | 800 | 200 | 0.81 |
| 5 | 800 | 200 | 0.85 |
Mean accuracy 0.846, standard deviation 0.027. The spread matters as much as the mean: a model scoring 0.85 ± 0.03 is more trustworthy than one scoring 0.86 ± 0.12.
Variants and when to use them
| Variant | How it splits | Use when |
|---|---|---|
| K-fold | Random equal folds | General-purpose default |
| Stratified k-fold | Preserves class proportions in each fold | Classification, especially imbalanced |
| Leave-one-out (LOOCV) | k equals the number of examples | Very small datasets; expensive, high variance |
| Group k-fold | Keeps all rows of a group together | Multiple rows per customer, patient or device |
| Time-series split | Trains on the past, validates on the future | Any time-ordered data |
| Repeated k-fold | Runs k-fold several times with different seeds | More stable estimates on small data |
| Nested CV | Inner loop tunes, outer loop evaluates | Tuning plus unbiased estimation |
The two most common mistakes
Random splits on time-series data. Training on data from after the validation period leaks the future into the past. Forecasting results look excellent and fail completely in production. Always split by time for time-dependent problems.
Ignoring groups. If one customer appears in 50 rows, random splitting puts some of their rows in training and others in validation. The model memorises that customer and scores well for the wrong reason. This is a form of data leakage. Use group-aware folds.
Preprocessing inside the loop
Scaling, imputation, encoding and feature selection must be fitted inside each fold, on the training part only. Fitting a scaler on the whole dataset before cross-validation leaks information from validation data into training and inflates the score. Pipelines in libraries such as scikit-learn exist to make this correct by default.
How many folds?
| k | Training data per fold | Cost | Variance of estimate |
|---|---|---|---|
| 3 | 67% | Low | Higher |
| 5 | 80% | Moderate | Good balance |
| 10 | 90% | Higher | Lower |
| n (LOOCV) | Almost all | Very high | Can be high |
Five or ten folds suit most problems. More folds mean more training data per fold, a slightly less pessimistic estimate, and proportionally more compute.
When not to use cross-validation
- Very large datasets: a single large validation split is already stable and far cheaper.
- Very expensive training: retraining an LLM k times is rarely justified.
- Strict temporal deployment: a rolling time-based evaluation often reflects reality better.
Limitations
- Multiplies training cost by k.
- Still optimistic if tuning is heavy and repeated on the same folds.
- Does not protect against a dataset that is unrepresentative of production.
- Requires careful splitting for grouped, nested or time-structured data.