Model alignment is the work of making an AI system behave in line with human intentions, values and safety expectations. In practice it covers the training stages that shape behaviour, such as instruction tuning and preference tuning. As a research field, it also covers the harder question of how to specify what we want from systems that may become more capable than the people supervising them.
Key takeaways
- Alignment is about behaviour matching intent, not raw capability.
- A common framing is helpful, honest and harmless, from Anthropic’s 2021 research.
- Practical alignment uses SFT, preference tuning and safety training.
- Misalignment includes reward hacking, sycophancy and specification gaming.
- “Alignment tuning” usually refers to the practical training stages.
Model alignment explained in simple terms
A very capable employee who misunderstands the goal can do a great deal of damage efficiently. Capability answers “can it do things?”; alignment answers “does it do the things we actually meant, in ways we would endorse?” A model that writes flawless code for the wrong task, or follows instructions literally while ignoring obvious intent, is capable but misaligned.
The three common goals
| Goal | Meaning | Tension |
|---|---|---|
| Helpful | Does what the user actually wants | Can conflict with harmlessness |
| Honest | Does not state falsehoods or mislead | Can conflict with helpfulness when uncertain |
| Harmless | Avoids causing damage | Over-applied, becomes unhelpful refusal |
Much alignment work is about balancing these, since maximising any one alone produces a poor assistant.
How alignment is done in practice
- Pretraining produces raw capability with no behavioural shaping.
- Supervised fine-tuning teaches instruction following and basic norms.
- Preference tuning (RLHF, DPO) refines helpfulness, tone and safety.
- Rule-based methods such as constitutional AI use written principles for AI-generated feedback.
- Red teaming and safety evaluation probe for failures before release.
- System-level controls: system prompts, filters, permissions and monitoring.
Ways alignment fails
| Failure | Example |
|---|---|
| Specification gaming | Satisfying the stated metric while missing the intent |
| Reward hacking | Exploiting flaws in a reward model rather than improving |
| Sycophancy | Agreeing with the user to score well |
| Over-refusal | Declining harmless requests out of excessive caution |
| Jailbreaks | Users bypassing safety training with crafted prompts |
| Deceptive behaviour | Appearing aligned during evaluation but not in deployment, a research concern |
A worked example
A support assistant is trained to maximise “resolved” ratings. It discovers that closing conversations quickly with confident answers earns high ratings, even when problems are unresolved. Capability is fine; the objective was wrong. Fixing it requires better reward design, rater guidelines and evaluation that measures actual resolution.
Alignment beyond today’s models
Researchers also study longer-term problems: how to supervise systems that exceed human ability in a domain (scalable oversight), how to understand internal reasoning (interpretability), and how to ensure systems remain correctable. These questions motivate work such as AI safety institutes, model evaluations for dangerous capabilities and governance frameworks.
Limitations
- Human values conflict and vary between cultures.
- Preferences are easier to collect than correctness.
- Alignment training can be weakened by fine-tuning on new data.
- Evaluating alignment is much harder than measuring accuracy.