Preference tuning is the training stage in which a model learns from comparisons between responses, rather than from single correct answers. People (or an AI judge) mark one response as better than another, and the model is trained to make preferred responses more likely. It usually follows supervised fine-tuning and is what turns a competent model into a genuinely helpful assistant.
Key takeaways
- Preference data consists of a prompt plus a chosen and a rejected response.
- Main methods are RLHF and DPO, with many variants.
- Judging which of two answers is better is far easier than writing the perfect answer.
- It improves helpfulness, tone, formatting and safety.
- Poorly designed preference data causes sycophancy and verbosity.
Preference tuning explained in simple terms
Teaching by example only goes so far. Imagine training a writer: showing them 50 good articles helps, but improvement accelerates when an editor says “this version is better than that one, and here’s the pair”. Preference tuning gives models that editorial signal at scale.
Why it is needed
Supervised fine-tuning teaches the model to imitate demonstrations. But many qualities that matter, such as being appropriately concise, refusing unsafe requests politely or choosing a helpful format, are hard to capture in demonstrations alone. Comparisons express them naturally.
Main approaches
| Method | How it learns | Notes |
|---|---|---|
| RLHF | Trains a reward model, then optimises with reinforcement learning | Powerful; complex and costly |
| DPO | Optimises directly on preference pairs | Simpler and more stable; widely used |
| RLAIF | Uses AI-generated preferences, often guided by written principles | Cheaper; quality depends on the judge |
| KTO | Uses simple good or bad labels rather than pairs | Easier data collection |
| ORPO | Combines supervised and preference learning in one step | Fewer stages |
| Reinforcement learning with verifiable rewards | Rewards come from automatic checks | Key for reasoning models |
A worked example
Prompt: “Can I take ibuprofen with my blood pressure medication?”
- Response A: “Yes, that’s fine.”
- Response B: “Ibuprofen can raise blood pressure and interact with some medications, so it’s worth checking with your pharmacist or doctor, who can look at your specific prescription.”
Raters prefer B. Across thousands of comparisons, the model learns to be careful with health questions, explain reasoning and point to appropriate help, rather than answering flatly.
Where preference data comes from
- Human raters comparing model outputs.
- Real user feedback such as thumbs up or down, used carefully.
- Expert comparisons for specialised domains.
- AI judges, validated against human agreement.
- Automatic checks where correctness can be verified.
Common pitfalls
| Pitfall | Cause |
|---|---|
| Sycophancy | Raters prefer agreement and flattery |
| Verbosity | Longer answers often score higher |
| Over-refusal | Safety preferences applied too broadly |
| Reward hacking | Model exploits flaws in the reward signal |
| Narrow preferences | A small rater pool encodes its own tastes |
Mitigations include length-controlled evaluation, diverse raters, clear rating guidelines and regular checks against held-out tasks.
Limitations
- Preference data is expensive and slow to collect well.
- Preferences differ between people and cultures.
- Optimising too hard can degrade other capabilities.
- Human preference is not the same as correctness.