Visual question answering (VQA) is the task of answering a natural-language question about an image. Given a photo of a shelf and the question “how many bottles are in the second row?”, a VQA system returns a text answer. It requires both computer vision and language understanding, and is now a standard capability of multimodal LLMs.
Key takeaways
- VQA answers a specific question; captioning describes the image generally.
- Modern VQA is handled by multimodal models rather than dedicated architectures.
- Question types vary from simple recognition to counting, reading and reasoning.
- Counting, spatial relations and small text remain the weak areas.
- Answers can be confidently wrong, so verification matters for business use.
VQA explained in simple terms
Captioning is someone describing a photograph. VQA is someone answering your questions about it: “is the door open?”, “what does the label say?”, “which shelf is empty?”. The second is far more useful in practice, because you ask only what you need.
How it works
- A vision encoder converts the image into visual tokens.
- The question is tokenised as text.
- A transformer attends across both together.
- The model generates the answer as text.
Research VQA began with fixed answer vocabularies, where models chose from thousands of possible short answers. Modern systems generate free-form text and can explain their reasoning.
Types of questions
| Question type | Example | Difficulty |
|---|---|---|
| Recognition | “What object is this?” | Usually reliable |
| Attributes | “What colour is the cable?” | Usually reliable |
| Reading | “What is the serial number?” | Depends on image quality |
| Counting | “How many boxes are there?” | Often unreliable |
| Spatial | “Is the valve left of the gauge?” | Frequently wrong |
| Comparison | “Which container is fuller?” | Mixed |
| Reasoning | “Is this safe to ship?” | Depends on domain knowledge |
| Absence | “Is anything missing?” | Hard |
A worked example
A field technician photographs an electrical panel and asks: “Is the main breaker on, and are any warning labels damaged?”
- The model identifies the breaker position and answers that it is on.
- It notes one warning label is partly torn.
- It cannot read a faded rating label, and says so.
That last behaviour matters: a good deployment instructs the model to state when it cannot determine something, rather than guessing.
Business uses
- Field service: diagnose equipment from technician photos.
- Insurance: answer structured questions about damage photos.
- Retail: check shelf compliance against planograms.
- Manufacturing: verify assembly steps from images.
- Support: interpret customer screenshots of errors.
- Document work: answer questions about scanned forms and charts.
Making VQA reliable
- Ask narrow, specific questions rather than open ones.
- Use structured output so answers are machine-checkable.
- Permit and expect “cannot determine” answers.
- Combine with detection for counting, which dedicated models do better.
- Provide reference context, such as the expected configuration.
- Sample and verify results before trusting automation.
Limitations
- Counting beyond a handful of objects is unreliable.
- Spatial reasoning such as left, right and behind is weak.
- Small or low-contrast text may be misread.
- Models may answer confidently from expectation rather than the image.
- Images can carry hidden instructions, a prompt injection risk.