AI bias, also called algorithmic bias, is systematic unfairness in an AI system’s outputs that disadvantages particular groups of people. It usually originates in the data, the problem framing or the deployment context rather than in the mathematics. This page covers fairness bias. For the statistical sense of bias in the bias-variance trade-off, see model bias.
Key takeaways
- Bias usually enters through historical data, under-representation or flawed proxies.
- Removing protected attributes does not remove bias, because proxies remain.
- Several mathematical fairness definitions exist, and they cannot all be satisfied simultaneously.
- Mitigation can happen before, during or after training.
- Measurement by group, on your own data, is the non-negotiable first step.
AI bias explained in simple terms
A model learns patterns from the past. If the past was unequal, the patterns encode that inequality, and the model reproduces it efficiently and at scale. The system is not malicious; it is faithfully reflecting data that reflected an unfair world, which is precisely why it needs deliberate checking.
Where bias comes from
| Source | Description | Example |
|---|---|---|
| Historical bias | The data reflects past discrimination | Past hiring favoured one group, so the model learns to |
| Representation bias | Some groups are under-represented | Face datasets with few darker-skinned faces |
| Measurement bias | Labels or features mean different things across groups | Arrest rates used as a proxy for crime rates |
| Aggregation bias | One model forced on groups that behave differently | A single clinical threshold across populations |
| Proxy variables | Neutral features correlate with protected traits | Postcode standing in for ethnicity |
| Deployment bias | The system is used differently from intent | A triage tool used as a final decision |
| Feedback loops | Model outputs shape future training data | Predictive policing concentrating patrols |
Documented cases
- Gender Shades (2018): Joy Buolamwini and Timnit Gebru found commercial gender classification systems had error rates under 1% for lighter-skinned men but up to around 35% for darker-skinned women.
- Recruiting tool (2018): Amazon abandoned an experimental hiring model after it was found to downgrade CVs associated with women.
- COMPAS (2016): investigative reporting found differences in error types between groups in a recidivism risk tool, prompting years of debate about which fairness definition should apply.
These cases are useful because each illustrates a different failure: data representation, historical labels, and conflicting fairness definitions.
Fairness metrics
| Metric | Requires | Trade-off |
|---|---|---|
| Demographic parity | Equal positive rates across groups | Ignores genuine base rate differences |
| Equal opportunity | Equal true positive rates | May leave false positive gaps |
| Equalised odds | Equal true and false positive rates | Hard to achieve with unequal base rates |
| Predictive parity | Equal precision across groups | Conflicts with equalised odds |
| Individual fairness | Similar individuals treated similarly | Requires defining similarity |
Research from 2016 and 2017 showed formally that when base rates differ between groups, several of these criteria cannot hold at once. There is therefore no purely technical answer: choosing a fairness criterion is a value judgement that should be made explicitly and documented.
A worked example
A hiring screen flags 20% of applicants for interview.
| Group | Applicants | Flagged | Flag rate |
|---|---|---|---|
| A | 1,000 | 240 | 24% |
| B | 1,000 | 160 | 16% |
The four-fifths rule used in some employment contexts compares 16% ÷ 24% = 0.67, below the 0.8 threshold, which would prompt investigation. Analysis shows “years at current employer” drives much of the gap, a feature correlated with career breaks. The team tests removing it, finds a negligible accuracy loss, and the gap narrows. The decision and its rationale are documented.
Mitigation approaches
| Stage | Techniques |
|---|---|
| Pre-processing | Rebalance or reweight data; improve representation; remove proxies |
| In-processing | Fairness constraints or penalties during training |
| Post-processing | Group-aware thresholds, where legally permitted |
| Process | Diverse review, documentation, human appeal routes |
Note that some mitigations, particularly group-specific thresholds, may be unlawful in certain jurisdictions. Legal advice matters.
Practical programme
- Decide which groups and outcomes to measure.
- Measure selection rates, error rates and calibration by group on your own data.
- Investigate causes, including proxies.
- Choose and document a fairness criterion.
- Apply mitigations and re-measure.
- Monitor continuously; bias can re-emerge as data drifts.
- Keep a human appeal route for affected people.
Limitations
- Fairness criteria conflict mathematically.
- Group data may be unavailable or sensitive to collect.
- Fixing one metric can worsen another.
- Technical fixes cannot resolve unfairness in the underlying process.