Explainable AI (XAI) is the set of methods and practices that make an AI system’s behaviour understandable to people. It answers questions such as “why was this loan declined?”, “which features drive this model’s predictions?” and “what would have changed the outcome?” XAI matters because many high-performing models, especially deep networks, are black boxes whose internal logic is not readable by inspection.
Key takeaways
- Local explanations explain one prediction; global explanations describe overall model behaviour.
- Common methods include SHAP, LIME, Grad-CAM, feature importance and counterfactuals.
- Some models are interpretable by design: linear models, decision trees, rule lists.
- Laws increasingly require meaningful explanation of automated decisions affecting people.
- An explanation that sounds convincing is not necessarily faithful to what the model actually did.
Explainable AI explained in simple terms
A doctor who says “you have condition X” without reasoning is hard to trust, question or correct. A doctor who says “given your test result, age and family history, these three factors point to X” lets you verify the logic and challenge it. XAI asks models to show their working in whatever way the audience can use.
Why explainability matters
| Reason | Example |
|---|---|
| Trust and adoption | Staff will not act on a risk score they cannot interrogate |
| Debugging | Explanations reveal when a model keys on a spurious feature |
| Fairness auditing | Checking whether proxies for protected traits drive decisions |
| Regulation | Credit, hiring and insurance decisions often require reasons |
| Recourse | Telling a person what they could change to get a different outcome |
| Safety | Understanding failure modes before deployment |
Local vs global explanations
Local explanations describe a single prediction: “this application was declined mainly because of a high debt-to-income ratio and three recent missed payments.” Global explanations describe the model overall: “income and credit utilisation account for most of the model’s behaviour; postcode has moderate influence.”
Both matter. Customers need local explanations. Risk and compliance teams need global ones.
The main XAI methods
| Method | Type | How it works | Best for |
|---|---|---|---|
| Coefficients | Global, built in | Read weights directly | Linear and logistic regression |
| Tree paths | Both, built in | Follow the split conditions | Decision trees, rule models |
| Feature importance | Global | Measure accuracy drop when a feature is shuffled | Any tabular model |
| SHAP | Local and global | Assigns each feature a contribution based on cooperative game theory (Shapley values), introduced for ML in 2017 | Tabular models, the current default |
| LIME | Local | Fits a simple model around one prediction, from 2016 research | Any model, quick local checks |
| Partial dependence / ICE | Global | Shows how predictions change as one feature varies | Understanding relationships |
| Grad-CAM | Local | Highlights image regions that drove a prediction | CNNs and vision models |
| Attention maps | Local | Shows which tokens were attended to | Transformers, with caution |
| Counterfactuals | Local | “If income were ₹2 lakh higher, the decision would flip” | Giving people actionable recourse |
| Mechanistic interpretability | Global, research | Reverse-engineers internal circuits | LLM safety research |
A worked example
A bank’s model declines a loan. A SHAP explanation for that application might show:
| Feature | Contribution to decline |
|---|---|
| Debt-to-income ratio 61% | +0.31 |
| Three missed payments in 12 months | +0.24 |
| Employment length 4 months | +0.11 |
| Credit history length 9 years | −0.08 |
| Requested amount | +0.02 |
The applicant can be told the three main reasons, and a counterfactual can add: “with no missed payments in the last year, this application would have been approved.” That is a useful, actionable explanation. It also lets the bank check that nothing resembling a protected characteristic is doing the work.
Explainability and the law
- GDPR requires meaningful information about the logic involved in solely automated decisions with legal or similarly significant effects, together with a right to human intervention.
- The EU AI Act requires high-risk systems to be sufficiently transparent for deployers to interpret output, with documentation and human oversight.
- Sector rules, such as credit regulations in several countries, require adverse action reasons.
- Explanations must be meaningful to the recipient, not just technically available.
Faithfulness vs plausibility
A critical distinction: an explanation can be plausible, sounding sensible to a human, without being faithful, actually reflecting the model’s computation. Attention weights, for example, show where a model looked, not why it concluded. Post-hoc explanations are approximations, and different methods can disagree on the same prediction. Where decisions are consequential, some researchers argue for using inherently interpretable models rather than explaining opaque ones after the fact.
Practical approach
- Decide who needs the explanation: customer, reviewer, auditor or engineer.
- Prefer an interpretable model if it performs comparably.
- Use SHAP for tabular models, Grad-CAM for vision, counterfactuals for recourse.
- Validate explanations: do they change sensibly when inputs change?
- Document limitations in a model card.
- Keep human review for decisions with significant impact.
Limitations
- Post-hoc explanations approximate rather than reveal.
- Methods can disagree, and some are unstable across runs.
- Explanations for very large models remain shallow.
- Explaining a model does not make its decision fair or correct.