Image captioning is the task of generating a natural-language description of an image. A vision encoder turns the picture into features, and a language decoder writes a sentence describing it. Captioning powers accessibility tools, image search, content management and alt text generation at scale.
Key takeaways
- Captioning combines computer vision with language generation.
- Early systems paired CNNs with RNNs; modern ones use multimodal LLMs.
- Prompting can control caption length, style and focus.
- Accessibility is the most valuable application.
- Models can describe plausible but absent details, a visual hallucination.
Image captioning explained in simple terms
Describing a photo over the phone to someone who cannot see it. You do not list pixels; you say “a woman in a yellow sari harvesting tea on a hillside in the morning light”. Captioning teaches machines to produce that kind of description automatically.
How it works
- A vision encoder, usually a CNN or vision transformer, converts the image into feature vectors.
- A projection layer maps those features into the language model’s embedding space.
- The language model generates text conditioned on those visual tokens.
- Decoding settings such as temperature shape the caption’s style and variety.
The landmark early approach, “Show and Tell” (2015), paired a CNN with an RNN. Contrastive image-text models such as CLIP (2021) improved how visual and textual meaning align, and modern multimodal models produce far richer, promptable descriptions.
Types of captions
| Type | Purpose | Example |
|---|---|---|
| Short caption | Quick summary | “A cat sleeping on a laptop” |
| Alt text | Accessibility | “Tabby cat asleep on an open silver laptop on a wooden desk” |
| Dense caption | Detailed description of regions | Multiple captions with locations |
| Domain-specific | Technical description | “Chest X-ray showing left lower lobe opacity” |
| Styled caption | Marketing or social copy | A product caption in brand voice |
A worked example
An e-commerce site has 40,000 product photos with no alt text, hurting accessibility and image search.
- Each image is captioned with a prompt: “Describe this product image factually in under 20 words, mentioning product type, colour and material.”
- Captions are checked against product metadata: if the stated colour is “navy” but the caption says “black”, the item is flagged.
- Approved captions become alt text; flagged ones go to review.
Grounding captions against known metadata is what makes automation safe at this scale.
Where captioning is used
- Accessibility: alt text and screen-reader descriptions.
- Search: making image libraries searchable by content.
- Content management: auto-tagging large media archives.
- Social and marketing: drafting post copy from visuals.
- Data labelling: generating descriptions for training datasets.
- Assistive apps: narrating surroundings for blind users.
Writing good alt text
Automated captions are a starting point, not a standard. Good alt text is concise, describes what matters in context, avoids “image of”, and omits decoration. A chart needs its data summarised, not its appearance described. Human review remains important for meaningful accessibility.
Limitations
- Models may state details that are not present.
- Cultural and contextual nuance is often missed.
- Text inside images may be misread; use OCR for that.
- Captions can reflect stereotypes present in training data.
- The same image needs different captions depending on where it appears.