Data leakage is when information that would not be available at prediction time finds its way into model training or evaluation. The result is a model that scores brilliantly in testing and fails in production. It is one of the most common and most expensive mistakes in applied machine learning, and it is dangerous precisely because the symptom, excellent validation performance, looks like success.
Key takeaways
- The test: would this information genuinely be available at the moment of prediction?
- Main types: target leakage, train-test contamination, temporal leakage, group leakage and preprocessing leakage.
- The classic warning sign is performance that seems too good.
- Prevention is structural: correct splits and pipelines, not vigilance alone.
- Leakage also affects LLM evaluation, where it is called benchmark contamination.
Data leakage explained in simple terms
A student finds the exam answers while revising. They score 98% and everyone concludes they understand the subject. In the next exam, with different questions, they fail. The high score was never a measure of knowledge; it measured access to answers. Leakage gives models that same unearned advantage.
The main types
| Type | What happens | Example |
|---|---|---|
| Target leakage | A feature encodes the answer or follows from it | “Account closed date” when predicting churn |
| Train-test contamination | The same or near-identical rows appear in both splits | Duplicated records, augmented copies split apart |
| Temporal leakage | Training uses data from after the prediction point | Random splits on time-series data |
| Group leakage | Rows from the same entity appear on both sides | One patient’s scans in both training and test |
| Preprocessing leakage | Statistics computed over all data before splitting | Fitting a scaler or imputer on the full dataset |
| Feature selection leakage | Features chosen using the whole dataset | Correlation screening before splitting |
| Tuning leakage | Repeated tuning against the test set | Treating the test set as a validation set |
A worked example
A team builds a model to predict which customers will cancel next month. Validation accuracy reaches 97%, far above the business’s expectations.
Investigation shows one feature is `support_tickets_last_7_days`, computed at the time the dataset was exported, which is after cancellation for churned customers. Cancelling customers raise tickets just before leaving, so the feature effectively encodes the outcome. At real prediction time, that window has not happened yet.
Rebuilt with features computed strictly as of the prediction date, accuracy falls to 79%. That lower number is the real one, and it is still useful. The first model would have failed silently in production.
Warning signs
- Accuracy far above what domain experts consider plausible.
- One feature dominating importance scores inexplicably.
- A sharp drop when the model is deployed or tested on newer data.
- Near-perfect scores on a problem known to be hard.
- Performance degrading when a single feature is removed.
Prevention by type
| Type | Prevention |
|---|---|
| Target leakage | For each feature, ask when the value becomes known; drop anything post-outcome |
| Contamination | Deduplicate before splitting, including near-duplicates |
| Temporal | Split by time; train on the past, test on the future |
| Group | Use group-aware splits in cross-validation |
| Preprocessing | Fit all transformations inside the training fold only, using pipelines |
| Feature selection | Perform selection inside the cross-validation loop |
| Tuning | Keep a test set untouched until the end |
Leakage in LLM evaluation
The same problem appears as benchmark contamination: if evaluation questions appeared in pretraining data, scores overstate ability. Teams address it by decontaminating training corpora, using held-out or newly created benchmarks, and testing on private evaluation sets. A model that scores far better on public benchmarks than on your own fresh test questions may be benefiting from contamination.
A practical checklist
- List every feature and when its value becomes known in reality.
- Deduplicate before splitting.
- Split by time for time-dependent problems, by group where entities repeat.
- Put all preprocessing inside a pipeline fitted on training data only.
- Reserve the test set and use it once.
- Have someone else review the feature list, since leakage is easier to spot from outside.
- Compare offline results with early production results and investigate large gaps.
Limitations
- Subtle leakage can survive careful review.
- Domain knowledge is needed to judge what is knowable when.
- Some leakage is only visible after deployment.
- Fixing it usually lowers reported accuracy, which can be politically awkward.