A multimodal LLM is a large language model that can take in, and sometimes produce, more than text, such as images, audio or video. It usually works by converting non-text inputs into token-like representations that the language model can reason over together with text. You can show it a photo, a chart or a scanned document and ask questions in plain language.
This page covers the model architecture. For the broader field of multimodal AI, including image and video generators, see What is Multimodal AI?
Key takeaways
- Multimodal LLMs combine an LLM with encoders for images, audio or video.
- A connector layer maps visual or audio features into the LLM’s embedding space.
- Some models are trained multimodal from the start; others add vision to an existing LLM.
- Common uses include document understanding, visual Q&A and voice assistants.
- Images use tokens too, so they count towards context window and cost.
A multimodal LLM explained in simple terms
A text-only LLM is like an expert on the phone: you must describe everything in words. A multimodal LLM is like an expert sitting next to you who can look at your screen, your receipt or your whiteboard sketch and hear your voice. It can combine what it sees and hears with what you say, just as a person would.
How a multimodal LLM works
- Modality encoders: a vision encoder, often a vision transformer, turns an image into feature vectors. Audio encoders do the same for sound.
- Projection or connector: a small network maps these features into the same embedding space the LLM uses for text tokens.
- Joint processing: the LLM’s transformer layers attend across image tokens and text tokens together.
- Output: the model generates text. Some “any-to-any” models can also output speech or images.
Two main design approaches
| Approach | How it works | Examples of the idea |
|---|---|---|
| Adapter-based | Connect a pretrained vision encoder to a pretrained LLM, then train the connector and fine-tune | LLaVA (2023) popularised this open approach |
| Natively multimodal | Train one model on text, images and audio from the start | Google’s Gemini (2023), OpenAI’s GPT-4o (2024) |
Natively multimodal models can respond faster in voice conversations and capture richer links between modalities, while adapter-based designs are cheaper to build.
A worked example: reading a chart
A user uploads a bar chart of quarterly sales and asks, “Which quarter grew fastest compared with the previous one?”
- The vision encoder splits the chart into patches and encodes them.
- The connector turns these into visual tokens.
- The LLM reads the axis labels and bar heights from the visual tokens, compares consecutive quarters and calculates growth.
- It answers: “Q3 grew fastest, up about 18% from Q2,” and explains its reasoning.
Accuracy depends on the chart’s clarity. For precise numbers, it is safer to also provide the underlying data.
What multimodal LLMs can do
- Read and extract data from documents, invoices, forms and screenshots.
- Answer questions about photos, diagrams and charts.
- Describe images for accessibility.
- Understand spoken questions and reply by voice.
- Analyse video frames and summarise clips.
- Help debug UI problems from screenshots.
Limitations
- May misread small text, dense tables or handwriting.
- Can hallucinate visual details that are not present.
- Weaker at precise counting and exact spatial measurements.
- Images and video consume many tokens, raising cost and latency.
- Images can carry hidden instructions, a prompt injection risk.
- Faces and personal documents raise privacy concerns.