Data labeling is the process of annotating raw data with the correct answers so that supervised learning models can learn from it. It covers tagging images, marking boxes around objects, transcribing audio, classifying text and writing ideal model responses. Label quality sets a ceiling on model quality, which is why experienced teams treat labelling as an engineering discipline rather than clerical work.
Key takeaways
- Models can only be as good as their labels.
- Clear guidelines matter more than annotator volume.
- Inter-annotator agreement measures whether the task is even well defined.
- Active learning and model assistance cut labelling cost substantially.
- Labels need maintenance as definitions and products change.
Data labeling explained in simple terms
Teaching someone to sort post requires showing them sorted examples. If your examples are inconsistent, with the same letter in different piles on different days, the learner internalises the confusion. Labelling is preparing those examples, and consistency matters more than quantity.
Types of labelling tasks
| Task | What annotators do | Effort per item |
|---|---|---|
| Classification | Assign a category | Seconds |
| Multi-label tagging | Apply several tags | Seconds to a minute |
| Object detection | Draw boxes around objects | Minutes |
| Segmentation | Outline objects pixel by pixel | Many minutes |
| Entity extraction | Mark names, dates, amounts in text | Minutes |
| Transcription | Write out spoken audio | Several times real-time |
| Ranking and preference | Choose the better of two outputs | Minutes |
| Instruction demonstrations | Write ideal responses for fine-tuning | Many minutes |
The labelling process
- Define the task precisely. Write a guideline document with definitions, examples and edge cases.
- Pilot. Have several annotators label the same 100 items.
- Measure agreement. Disagreement usually means unclear guidelines, not careless annotators.
- Revise guidelines and repeat until agreement is acceptable.
- Scale up with quality controls in place.
- Audit continuously with gold-standard items mixed into the queue.
- Maintain as categories and products evolve.
The most common mistake is skipping steps 2 to 4 and discovering months later that 30% of labels are inconsistent.
Measuring label quality
| Method | What it tells you |
|---|---|
| Inter-annotator agreement (Cohen’s or Fleiss’ kappa) | Whether the task is well defined |
| Gold standard items | Whether individual annotators are accurate |
| Consensus labelling | Majority vote across annotators reduces noise |
| Expert adjudication | Resolves hard cases consistently |
| Review sampling | Ongoing quality monitoring |
A kappa above roughly 0.8 indicates strong agreement; below 0.6 usually means the task needs redefinition rather than more training.
Reducing cost
| Technique | How it helps |
|---|---|
| Active learning | Label only the examples the model is most uncertain about |
| Model-assisted labelling | A model pre-labels; humans correct, which is far faster than labelling from scratch |
| LLM-assisted labelling | Use a strong model to draft labels, with human review on a sample |
| Weak supervision | Combine noisy heuristics and rules to produce probabilistic labels |
| Transfer learning | Start from a pre-trained model so fewer labels are needed |
| Prioritisation | Label where errors are costliest first |
A common and effective pattern is to label a small high-quality seed set, train a model, use it to pre-label, and have humans correct its output.
A worked example
A company needs 20,000 labelled support tickets across eight categories.
- Manual labelling at 20 seconds each: roughly 111 hours.
- With an LLM pre-labelling and humans correcting at 6 seconds each: roughly 33 hours, plus review of a sample of the model’s confident labels.
- Agreement testing on 200 tickets first reveals that two categories overlap; merging them raises kappa from 0.58 to 0.84 and removes a large share of disagreements before scaling.
The time spent on step one saved more than it cost.
Ethical considerations
Much large-scale labelling is outsourced, sometimes including exposure to disturbing content for moderation datasets. Fair pay, reasonable workloads, psychological support for harmful content and transparency about working conditions are part of responsible AI practice, not optional extras.
Limitations
- Expensive and slow for complex tasks.
- Ambiguous tasks cap achievable agreement and therefore model accuracy.
- Labels encode the labellers’ assumptions, which can introduce bias.
- Labels decay as categories and real-world distributions change.