RLHF (reinforcement learning from human feedback) is a training method that uses human preferences to make AI models more helpful, honest and safe. People compare pairs of model responses and choose the better one. A reward model learns to predict these preferences, and the language model is then trained with reinforcement learning to produce responses that score highly. RLHF was central to making chat assistants such as ChatGPT feel useful.
Key takeaways
- RLHF has three stages: supervised fine-tuning, reward model training and reinforcement learning.
- Humans rank or compare outputs rather than writing perfect answers.
- A KL penalty keeps the model from drifting too far from its starting point.
- It improves helpfulness, tone and safety, but can cause sycophancy and reward hacking.
- Simpler alternatives such as DPO skip the separate reward model and RL loop.
RLHF explained in simple terms
Writing the perfect answer to every possible question is impossible. But judging which of two answers is better is easy. RLHF uses this. Imagine a chef who cooks two versions of a dish; tasters pick the one they prefer each time. A food critic learns the tasters’ preferences, and the chef keeps adjusting recipes to impress the critic. The tasters are human labellers, the critic is the reward model, and the chef is the language model.
The three stages of RLHF
- Supervised fine-tuning: fine-tune a pretrained model on example prompts and good responses written by people.
- Reward model training: generate several responses to many prompts; human labellers rank them. Train a reward model that takes a prompt and response and outputs a score predicting how much people would like it.
- Reinforcement learning: the language model generates responses, the reward model scores them, and an algorithm, often PPO (proximal policy optimisation), updates the model to increase reward. A KL penalty discourages the model from moving too far from the supervised model, which helps prevent strange or degenerate outputs.
A worked example
Prompt: “My laptop won’t turn on. What should I do?”
- Response A: “Buy a new laptop.”
- Response B: “Try these steps: check the charger and power light, hold the power button for 15–30 seconds, remove any connected devices, and try a different outlet. If it still won’t start, contact the manufacturer’s support.”
Labellers consistently prefer B. After thousands of such comparisons, the reward model learns that specific, safe, step-by-step help scores well. RL then shifts the language model towards producing answers like B across many topics.
Key milestones
| Year | Work | Contribution |
|---|---|---|
| 2017 | Paul Christiano and colleagues | Deep RL from human preferences for games and robotics |
| 2020 | OpenAI summarisation research | Applied human preferences to language model summaries |
| 2022 | InstructGPT (Ouyang et al.) | Showed people preferred a 1.3-billion-parameter RLHF model over 175-billion-parameter GPT-3 |
| 2022 | Anthropic’s helpful and harmless research | Studied trade-offs between helpfulness and harmlessness |
Variants and alternatives
- RLAIF: AI feedback replaces some human feedback, often guided by written principles.
- DPO: optimises directly on preference pairs without training a separate reward model.
- Reinforcement learning with verifiable rewards: rewards come from automatic checks, such as correct maths answers or passing code tests, which is key for reasoning models.
Limitations
- Cost: collecting high-quality human comparisons is expensive.
- Sycophancy: models may learn to agree with users or flatter because people rate that highly.
- Reward hacking: the model exploits flaws in the reward model rather than truly improving.
- Labeller bias: preferences reflect who the labellers are and their instructions.
- Complexity: RL training is unstable and harder to run than supervised fine-tuning.