Model distillation, also called knowledge distillation, is a technique that trains a smaller “student” model to imitate a larger, more capable “teacher” model. The student learns from the teacher’s outputs rather than only from original labelled data. The result is a model that is much cheaper and faster to run while keeping a large share of the teacher’s performance.
Key takeaways
- Distillation transfers knowledge from a big model to a small one.
- Students often learn from the teacher’s full probability outputs, called soft labels, which carry more information than single correct answers.
- For LLMs, distillation often means training on responses generated by a stronger model.
- It powers many efficient production and on-device models.
- Using another company’s model outputs to train a competitor may breach terms of service.
Model distillation explained in simple terms
A senior expert cannot sit in every meeting, so they train a junior colleague. The junior does not re-learn everything from scratch; they watch how the expert handles thousands of cases, including how confident the expert is between options. Eventually the junior handles most cases well on their own, faster and at lower cost. The expert is the teacher model; the junior is the student.
How distillation works
- Choose a teacher: a large, accurate model.
- Choose a student: a smaller architecture, often with far fewer parameters.
- Generate teacher outputs on a large set of inputs.
- Train the student to match the teacher’s outputs, often combined with the original correct labels.
- Evaluate the student against the teacher and deploy it where speed and cost matter.
Soft labels: why they help
Geoffrey Hinton, Oriol Vinyals and Jeff Dean popularised modern knowledge distillation in 2015. Their key idea was to train on the teacher’s full probability distribution, softened with a temperature setting.
Worked example: a teacher classifying an image might output: dog 0.80, wolf 0.15, cat 0.05. A hard label just says “dog”. The soft label also teaches that this image looks somewhat like a wolf and very little like a cat. This “dark knowledge” about similarities helps the student learn more from each example.
Distillation for LLMs
| Approach | How it works |
|---|---|
| Logit distillation | The student matches the teacher’s token probabilities; requires access to the teacher’s internals |
| Response (sequence) distillation | The teacher generates answers; the student is fine-tuned on them |
| Reasoning distillation | The student learns from a reasoning model’s step-by-step solutions |
| On-policy distillation | The student generates answers and the teacher scores or corrects them |
Examples: DistilBERT (2019) was reported to be 40% smaller and 60% faster than BERT while keeping about 97% of its language understanding performance. More recently, several labs released smaller models distilled from their larger reasoning models, bringing strong reasoning to models that fit on a single GPU.
Distillation vs quantization vs fine-tuning
| Technique | What changes | Main benefit |
|---|---|---|
| Distillation | A new, smaller model is trained | Much cheaper model with inherited ability |
| Quantization | Same model, fewer bits | Less memory, faster inference |
| Fine-tuning | Existing model adapted to a task | Better task performance |
These can be combined: distil a small model, fine-tune it for a task, then quantize it for deployment.
Legal and ethical points
Many AI providers’ terms prohibit using their outputs to develop competing models. Distilling from open models with permissive licences, or from your own models, avoids this issue. Students also inherit their teacher’s errors and biases.
Limitations
- The student rarely matches the teacher on every task, especially rare or complex ones.
- Requires a strong teacher and plenty of data or compute to generate outputs.
- Teacher mistakes and biases transfer to the student.
- Terms of service may restrict which teachers you can use.