Image segmentation is the computer vision task of labelling every pixel in an image, producing exact object outlines rather than rectangular boxes. Where object detection says “there is a tumour roughly here”, segmentation says “these precise pixels are tumour”. That precision matters in medical imaging, manufacturing measurement, satellite analysis and photo editing.
Key takeaways
- Output is a mask: a label for every pixel.
- Semantic segmentation labels categories; instance separates individual objects; panoptic does both.
- U-Net (2015) and Mask R-CNN (2017) are landmark architectures.
- Segment Anything (2023) enabled promptable segmentation without task-specific training.
- Annotation is the most expensive part, since masks must be drawn by hand.
Segmentation explained in simple terms
Detection puts a rectangle around a cat, which includes background in the corners. Segmentation traces the cat’s outline exactly: ears, tail, whiskers. If you need to measure area, cut the object out, or know exactly where a boundary lies, you need the outline, not the box.
Types of segmentation
| Type | What it produces | Example |
|---|---|---|
| Semantic | Every pixel labelled by class, objects not separated | All “road” pixels, all “car” pixels |
| Instance | Separate mask per object, background ignored | Car 1, car 2, car 3 individually |
| Panoptic | Both: classes for everything, instances for countable things | Road and sky as regions, each car separately |
| Binary | Foreground versus background | Product versus conveyor belt |
How segmentation models work
Most use an encoder-decoder shape. The encoder progressively reduces the image into features; the decoder upsamples back to full resolution, producing a label for every pixel. Skip connections carry fine detail from early layers to the decoder, which is essential for sharp boundaries. U-Net, designed for biomedical images in 2015, established this pattern and remains widely used.
Mask R-CNN (2017) extended object detection by adding a mask branch, giving instance segmentation. More recently, Segment Anything (Meta, 2023) introduced promptable segmentation: a click, box or point produces a mask for almost any object, without training for that specific class.
A worked example
A solar company measures usable roof area from aerial photographs.
- Detection would give a box around each roof, including surrounding garden.
- Semantic segmentation labels every roof pixel, so area can be calculated directly.
- Excluding pixels labelled as chimneys, vents and shaded regions gives usable area.
- Multiplying pixel counts by the image’s ground resolution converts this to square metres.
Only pixel-level labelling makes this measurement possible.
Metrics
| Metric | Meaning |
|---|---|
| IoU per class | Overlap between predicted and true masks |
| mIoU | Mean IoU across classes, the standard benchmark |
| Dice coefficient | Overlap measure common in medical imaging |
| Pixel accuracy | Share of correctly labelled pixels; misleading with imbalance |
| Boundary metrics | How accurate the edges are specifically |
Practical considerations
- Annotation cost: drawing masks can take minutes per image; promptable models and semi-automatic tools reduce this considerably.
- Resolution: fine structures need higher resolution, which raises compute cost.
- Class imbalance: small objects contribute few pixels, so loss functions often weight them.
- Boundary quality: overall IoU can look good while edges are poor.
Limitations
- Expensive to label and to run compared with classification.
- Thin structures and transparent objects are hard.
- Ambiguous boundaries, such as shadows or blur, reduce agreement even between human annotators.
- Video segmentation adds temporal consistency requirements.