AI benchmarking is the practice of measuring model performance on standardised tests so that different systems can be compared. Benchmarks drive research progress, inform purchasing decisions and shape marketing claims. They are also widely misread: a model that tops a leaderboard may perform poorly on your specific task, and published scores can be inflated by contamination.
Key takeaways
- Public benchmarks measure general capability, not fitness for your use case.
- Contamination occurs when benchmark data appears in training data, inflating scores.
- Saturation occurs when top models cluster near the ceiling and the benchmark stops discriminating.
- Human preference leaderboards measure something different from knowledge tests.
- Your own evaluation set, built from real tasks, is more useful than any public score.
AI benchmarking explained in simple terms
Exam results tell you something about a candidate, but not whether they will do well in your specific job. Benchmarks are the exams: standardised, comparable and useful, yet never a substitute for a work sample test. The organisations getting the most from AI run their own work sample tests.
Categories of benchmark
| Category | Measures | Examples |
|---|---|---|
| Knowledge and reasoning | Breadth of knowledge, multi-step reasoning | MMLU, MMLU-Pro, GPQA |
| Maths | Arithmetic and problem solving | GSM8K, MATH |
| Coding | Code generation and repair | HumanEval, SWE-bench |
| Long context | Retrieval and reasoning over long inputs | Needle-in-a-haystack style tests |
| Safety | Refusals, harmful content, robustness | Red-team and safety suites |
| Holistic frameworks | Many dimensions together | HELM |
| Human preference | Which response people prefer | Pairwise arena-style rankings |
| Agentic | Multi-step tool use and task completion | Task-based agent benchmarks |
Four problems with benchmarks
1. Contamination. If benchmark questions appeared in pretraining data, the model may recall rather than reason. This is data leakage at internet scale. Mitigations include decontamination, private held-out sets and continuously refreshed questions.
2. Saturation. When the best models all score above 90%, remaining differences are mostly noise and label errors. Several once-standard benchmarks have been retired or replaced for this reason.
3. Construct mismatch. A multiple-choice knowledge test measures something quite different from drafting a customer email correctly, following a format, or avoiding confident errors on your documents.
4. Goodhart’s law. Once a benchmark becomes a target, it stops being a good measure. Optimising for leaderboards can crowd out work on qualities users actually notice.
Reading benchmark results carefully
- Check the exact evaluation setup: prompting method, number of examples, whether reasoning was enabled.
- Check who ran it: self-reported results use the most favourable configuration.
- Check the date: scores shift as models and benchmarks change.
- Compare relative gaps, not decimals: a two-point difference rarely matters in practice.
- Look for contamination statements in technical reports.
Building your own evaluation set
This is where the real value lies.
- Collect 50–200 real tasks from your users or workflows.
- Write the correct or acceptable outputs, or the criteria for acceptability.
- Include hard cases, edge cases and cases where the right answer is “I don’t know”.
- Score automatically where possible: exact match, schema validity, retrieval recall.
- Use a model as a judge for subjective quality, validated against human ratings on a sample.
- Keep the set private, so it cannot leak into any model’s training data.
- Re-run it whenever you change model, prompt or pipeline.
A private set of 100 real tasks will tell you more about a model’s suitability than every public leaderboard combined.
A worked example
A support team compares three models.
| Model | Public reasoning benchmark | Internal eval: correct and grounded answers |
|---|---|---|
| A | 86% | 71% |
| B | 82% | 84% |
| C | 88% | 67% |
Model C leads publicly and performs worst internally, because the internal set rewards grounding in company documents and penalises confident invention. The public ranking was actively misleading for this use case.
Limitations
- Benchmarks measure averages, not failure modes that matter to you.
- Contamination is hard to rule out entirely.
- Human preference rankings can reward style over accuracy.
- Scores change as models are updated, so comparisons expire quickly.