DPO (direct preference optimization) is a method for aligning language models with human preferences by training directly on pairs of responses, one preferred and one rejected. Unlike RLHF, it does not need a separate reward model or a reinforcement learning loop. It was introduced in 2023 by Rafael Rafailov and colleagues at Stanford, and quickly became a popular, simpler alternative for preference tuning.
Key takeaways
- DPO uses the same kind of preference data as RLHF: a prompt, a chosen response and a rejected response.
- It turns preference learning into a simple supervised-style training objective.
- It is cheaper, more stable and easier to implement than classic RLHF.
- A setting called beta controls how far the model may move from its starting point.
- Many open-weight models have used DPO or its variants in post-training.
DPO explained in simple terms
RLHF is like hiring a food critic (the reward model) to learn what diners like, and then having the chef cook to please the critic. DPO skips the critic. The chef directly studies pairs of dishes where diners chose one over the other and adjusts recipes to make the preferred kind more likely and the rejected kind less likely. Fewer moving parts, similar goal.
How DPO works
- Start with a supervised fine-tuned model. Keep a frozen copy as the reference model.
- Collect preference data: for each prompt, a chosen response and a rejected response, labelled by people or by an AI judge.
- Train: for each pair, increase the model’s likelihood of the chosen response and decrease that of the rejected response, measured relative to the reference model.
- Control drift with the beta parameter, which limits how far the model moves from the reference.
The key insight in the paper, titled “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, is that the optimal RLHF solution can be written in terms of the language model itself, so a separate reward model is unnecessary.
A worked example
Prompt: “Explain GST to a small shop owner in two sentences.”
- Chosen: “GST is a tax added to the price of most goods and services you sell. You collect it from customers and pay it to the government, and you can reduce what you owe using GST already paid on your business purchases.”
- Rejected: “GST, or Goods and Services Tax, is a comprehensive, multi-stage, destination-based indirect tax levied on every value addition under a dual structure with CGST, SGST and IGST components…”
DPO nudges the model to make answers like the first, which are clear and follow the two-sentence instruction, more likely, and answers like the second less likely. Over thousands of pairs, the model learns the preferred style.
DPO vs RLHF
| Aspect | RLHF | DPO |
|---|---|---|
| Reward model | Required | Not required |
| Training method | Reinforcement learning, often PPO | Supervised-style loss on preference pairs |
| Stability | Can be unstable | More stable |
| Compute and complexity | Higher | Lower |
| Online exploration | Model generates new responses during training | Usually uses a fixed, offline dataset |
| Typical use | Large frontier post-training pipelines | Widely used, especially for open models |
Related methods
Researchers have proposed many variants, such as IPO, KTO (which uses simple “good” or “bad” labels instead of pairs) and ORPO (which combines fine-tuning and preference learning in one step). Some labs combine DPO with online data generation to capture some benefits of RL.
Limitations
- Quality depends entirely on the preference data.
- Offline data may not cover the mistakes the current model actually makes.
- Can over-optimise, for example making answers longer if longer responses were often preferred.
- Does not add knowledge; it shapes behaviour.