Image classification is the computer vision task of assigning a label to an entire image. Given a photograph, the model outputs a category and a confidence score: defective or acceptable, cat or dog, invoice or receipt. It is the simplest vision task and the foundation for most others, since detection and segmentation build on the same feature-learning approach.
Key takeaways
- The output is a label for the whole image, not object locations.
- Most models are CNNs or vision transformers, usually fine-tuned from pre-trained weights.
- Single-label, multi-label and hierarchical setups suit different problems.
- Accuracy alone is misleading when classes are imbalanced.
- Transfer learning makes good results achievable with modest datasets.
Image classification explained in simple terms
Sorting a box of photographs into labelled piles. You are not marking where the cat is in the picture or tracing its outline; you are deciding which pile the whole photograph belongs to. That is classification.
How it works
- Collect labelled images covering each class and real-world conditions.
- Split into training, validation and test sets.
- Start from a pre-trained model and fine-tune on your data.
- Apply augmentation such as flips, crops, rotation and brightness changes.
- Train with cross-entropy loss; the final layer produces class probabilities.
- Choose a confidence threshold based on the cost of each error type.
Types of classification
| Type | Setup | Example |
|---|---|---|
| Binary | Two classes | Defective or acceptable |
| Multi-class | One label from many | Document type: invoice, receipt, contract |
| Multi-label | Several labels at once | A photo tagged “outdoor”, “people”, “night” |
| Hierarchical | Nested categories | Vehicle → car → hatchback |
| Fine-grained | Very similar classes | Bird species, product variants |
A worked example
A factory classifies tile images as “cracked” or “sound”.
- 4,000 labelled images, 8% cracked.
- A model predicting “sound” every time scores 92% accuracy and is useless.
- The team therefore measures recall on cracked tiles: what share of real cracks are caught.
- At a 0.4 confidence threshold, recall reaches 96% with 7% false alarms; a human checks flagged tiles.
- Lowering the threshold catches more cracks at the cost of more reviews, which is a business decision, not a technical one.
Metrics that matter
| Metric | Use |
|---|---|
| Accuracy | Only meaningful with balanced classes |
| Precision | How many flagged items really belong to the class |
| Recall | How many real cases are caught |
| F1 | Balance of precision and recall |
| Confusion matrix | Which classes get mixed up |
| Top-5 accuracy | Common in large-class benchmarks |
Transfer learning
Training from scratch needs very large datasets. In practice, teams fine-tune a model pre-trained on a large general dataset, which already knows edges, textures and shapes. A few hundred to a few thousand labelled images per class is often enough for strong results, which is why classification is usually the cheapest vision task to deploy.
Common pitfalls
- Shortcut learning: the model keys on backgrounds, lighting or watermarks rather than the object.
- Class imbalance: rare classes are ignored without weighting or resampling.
- Data leakage: near-duplicate images across splits inflate test scores.
- Distribution shift: a new camera, lens or lighting setup degrades accuracy.
- Ambiguous labels: inconsistent human labelling caps achievable accuracy.
Limitations
- Provides no location information; use object detection for that.
- Assumes one dominant subject per image.
- Struggles with classes not seen in training.
- Confidence scores are often poorly calibrated.