Data curation for LLMs is the process of selecting, cleaning, filtering, deduplicating and balancing the text used to train a model. Since a model can only learn from what it is shown, curation has as much influence on final quality as architecture or model size. Modern training teams often spend more effort on data than on model design.
Key takeaways
- Quality and diversity matter more than raw volume.
- Deduplication improves quality and reduces memorisation.
- Decontamination removes benchmark data so evaluations stay honest.
- Data mixture weights decide how much of each domain the model sees.
- Curation also covers privacy, licensing and safety filtering.
Data curation explained in simple terms
Training data is a diet. Feeding a model everything on the internet is like eating whatever is nearest: some of it is nourishing, much of it is junk, and some is harmful. Curation is planning the diet: choosing good sources, removing spoiled items, avoiding the same meal repeatedly, and balancing the food groups.
The curation pipeline
| Stage | What it does |
|---|---|
| Collection | Gather web crawls, books, code, papers and licensed data |
| Language identification | Detect and route by language |
| Text extraction | Strip HTML, menus, adverts and boilerplate |
| Quality filtering | Remove gibberish, spam and machine-translated junk with rules and classifiers |
| Deduplication | Remove exact and near-duplicate documents and passages |
| Safety filtering | Remove illegal and harmful content |
| Privacy filtering | Remove or mask personal data where possible |
| Decontamination | Remove text overlapping with evaluation benchmarks |
| Mixing and weighting | Decide the share of each source in the final corpus |
| Ordering | Choose the sequence of data seen during training |
Why deduplication matters
Web corpora contain enormous repetition: the same article republished across sites, boilerplate legal text, repeated templates. Research has found that deduplicating training data improves model quality, reduces memorisation of specific passages, and saves compute. Both exact matching and near-duplicate detection, often using hashing methods, are used.
A worked example: data mixing
A team building a coding assistant might target a mixture such as:
| Source | Share |
|---|---|
| High-quality web text | 45% |
| Source code | 30% |
| Technical documentation | 10% |
| Books and long-form text | 10% |
| Maths and reasoning data | 5% |
Raising the code share improves programming ability but can weaken general writing. These weights are usually tuned by training small models on candidate mixtures and comparing results, since full-scale experiments are too expensive to repeat.
Quality filtering approaches
- Heuristics: document length, punctuation ratios, symbol counts, repeated lines.
- Classifier-based: train a model to recognise text resembling high-quality references.
- Perplexity filtering: drop documents a reference model finds extremely unusual.
- Model-based judging: use an LLM to rate educational value, as used in open datasets such as FineWeb-Edu (2024).
Decontamination
If benchmark questions appear in training data, evaluation scores become meaningless. Teams search for overlaps with common benchmarks and remove them. This is imperfect: paraphrased or translated versions can slip through, which is one reason benchmark results should be treated with caution.
Curation for fine-tuning
For supervised fine-tuning and preference data, curation shifts from volume to precision: a few thousand carefully checked examples often beat tens of thousands of noisy ones. Checks include correctness, format consistency, diversity of tasks and removal of examples that teach bad habits.
Limitations
- Filters can remove valuable minority-language or specialised content.
- Quality classifiers encode their creators’ judgements.
- Full decontamination is difficult to guarantee.
- Curation at web scale is computationally expensive.
- Licensing status of much web text remains legally contested.