Model collapse is the progressive degradation that occurs when AI models are repeatedly trained on data generated by earlier AI models. Each generation reproduces the centre of the previous distribution more strongly and its edges more weakly, so rare cases, unusual phrasing and minority patterns gradually disappear. The concern became prominent as the public web filled with AI-generated text and images that future models may train on.
Key takeaways
- Collapse happens when generated data replaces real data across training generations.
- Early collapse loses the tails; late collapse converges on narrow, bland output.
- Research published in Nature in 2024 demonstrated the effect experimentally.
- Mixing in real data substantially reduces the risk.
- It does not mean all synthetic data is harmful; curated, verified synthetic data remains valuable.
Model collapse explained in simple terms
Photocopy a photograph, then photocopy the copy, then copy that. Each generation loses detail that was faint to begin with. Within a few rounds the subtle shading is gone and only the bold outlines remain. Models trained on model output behave similarly: what was rare in generation one becomes rarer in generation two and absent by generation five.
Why it happens
- A model learns an approximation of the real data distribution, strongest where data is dense.
- When it generates, it under-represents the tails: unusual words, rare facts, minority styles.
- If that output becomes training data, the next model learns the narrowed distribution as if it were reality.
- Sampling and filtering choices compound the narrowing.
- Repeated across generations, variance shrinks and errors accumulate.
Two stages
| Stage | What is lost | Visible symptom |
|---|---|---|
| Early collapse | The tails of the distribution | Rare facts, dialects and unusual constructions disappear |
| Late collapse | Most variety | Repetitive, homogeneous, low-quality output |
Early collapse is the more insidious because overall benchmark scores can look fine while coverage of rare but important cases quietly erodes.
The evidence
A 2024 study published in Nature trained successive generations of models on their predecessors’ outputs and documented clear degradation, with low-probability events vanishing first. Follow-up research found that the effect is driven largely by replacing real data. When each generation’s synthetic data is added to the original real data rather than substituted for it, degradation is far milder or absent.
A worked example
A team generates product descriptions with a model, publishes them, and later scrapes its own site to build training data.
- Generation 1 writes competent descriptions with varied sentence structures.
- Generation 2, trained partly on that output, favours the most common phrasings.
- Generation 3 produces noticeably formulaic copy, and unusual product categories are described generically.
- Benchmarks on common categories barely move, so the problem is invisible until someone reviews niche products.
Prevention
| Measure | Why it helps |
|---|---|
| Keep real data in the mix | Anchors the distribution to reality |
| Verify synthetic data | Keep only outputs that pass objective checks |
| Filter and deduplicate | Prevents amplification of repeated patterns |
| Track provenance | Know which training data is model-generated |
| Preserve archival datasets | Pre-AI-era corpora remain valuable references |
| Evaluate on rare cases | Measure tail performance, not just averages |
| Use diverse generators | Avoid training solely on one model’s style |
Does this mean synthetic data is bad?
No. Carefully constructed synthetic data has produced strong results, particularly when correctness can be verified, such as maths with checkable answers or code that must pass tests. The danger is unfiltered recursive training: scraping model output indiscriminately and treating it as fresh human data. Verification and real-data anchoring are what separate useful synthetic data from collapse.
Wider implications
- Web-scale datasets increasingly contain AI-generated content, making provenance harder.
- Content authenticity standards and watermarking help identify synthetic material.
- Human-written, verified data may become more valuable over time.
- Teams fine-tuning on their own model’s outputs should be especially careful.
Limitations of current understanding
- Most evidence comes from controlled experiments, not web-scale training.
- Real pipelines filter heavily, which changes the dynamics.
- The practical threshold of synthetic content that causes harm is not well established.
- Effects may differ between text, image and audio models.