A confusion matrix is a table showing how a classification model’s predictions compare with the true labels. For a binary problem it has four cells: true positives, false positives, true negatives and false negatives. Nearly every classification metric, including precision, recall and F1, is derived from these four numbers, which is why reading a confusion matrix is the foundation of evaluating a classifier.
Key takeaways
- The four cells distinguish which kind of error a model makes, not just how many.
- Accuracy alone hides the difference between false positives and false negatives.
- The matrix changes with the decision threshold, so it describes one operating point.
- For multi-class problems it becomes an n × n grid showing which classes get confused.
- Business impact depends on the cost of each error type, which the matrix makes visible.
The four outcomes
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP): correctly caught | False negative (FN): missed |
| Actually negative | False positive (FP): false alarm | True negative (TN): correctly ignored |
In statistics, a false positive is a Type I error and a false negative a Type II error.
A worked example
A fraud model is evaluated on 10,000 transactions, of which 200 are genuinely fraudulent.
| Predicted fraud | Predicted legitimate | |
|---|---|---|
| Actually fraud | TP = 150 | FN = 50 |
| Actually legitimate | FP = 300 | TN = 9,500 |
From these four numbers:
- Accuracy = (150 + 9,500) ÷ 10,000 = 96.5%
- Precision = 150 ÷ (150 + 300) = 33.3%
- Recall = 150 ÷ (150 + 50) = 75%
- Specificity = 9,500 ÷ (9,500 + 300) = 96.9%
- F1 = 2 × (0.333 × 0.75) ÷ (0.333 + 0.75) = 0.46
Accuracy of 96.5% sounds excellent. The matrix tells the real story: only a third of flagged transactions are actually fraud, and a quarter of real fraud is missed. A model predicting “legitimate” always would score 98% accuracy and catch nothing.
Turning cells into business decisions
Attach a cost to each error type:
| Outcome | Business consequence | Estimated cost |
|---|---|---|
| TP | Fraud stopped | Saves ₹8,000 on average |
| FP | Genuine customer blocked | ₹500 in support and goodwill |
| FN | Fraud succeeds | ₹8,000 loss |
| TN | Normal transaction | ₹0 |
With these numbers: value = (150 × 8,000) − (300 × 500) − (50 × 8,000) = ₹1,200,000 − ₹150,000 − ₹400,000 = ₹650,000 saved. Lowering the threshold would catch more fraud but create more false alarms; the matrix plus costs tells you whether that trade is worth making.
Multi-class confusion matrices
For n classes, the matrix is n × n, with rows as true classes and columns as predictions. The diagonal holds correct predictions; everything off-diagonal shows which classes get mixed up. This is diagnostically rich: discovering that “billing” tickets are mostly misclassified as “account” tells you exactly where to improve guidelines or training data.
Thresholds and the matrix
A classifier outputs probabilities; a threshold turns them into decisions. Every threshold produces a different confusion matrix:
| Threshold | TP | FP | FN | Effect |
|---|---|---|---|---|
| 0.3 | 180 | 700 | 20 | Catches more, more false alarms |
| 0.5 | 150 | 300 | 50 | Balanced |
| 0.8 | 90 | 60 | 110 | Few false alarms, misses much fraud |
Choosing the threshold is a business decision, informed by these numbers rather than by a single metric.
Practical guidance
- Always inspect the matrix, not just headline accuracy.
- Report absolute counts alongside percentages so rare classes are visible.
- Produce separate matrices for important subgroups to check for bias.
- Normalise by row to see per-class recall, by column to see per-class precision.
- Re-evaluate after deployment, since class balance shifts over time.
Limitations
- Describes a single threshold, not overall ranking quality.
- Large multi-class matrices are hard to read without visualisation.
- Says nothing about confidence calibration.
- Needs reliable ground-truth labels, which can themselves be wrong.