LLM inference is the process of running a trained large language model to generate output from a prompt. It has two phases: prefill, where the model processes the whole prompt at once, and decode, where it generates the response one token at a time. Inference determines how fast responses appear, how many users a system can serve and how much each request costs.
This page covers inference specifically for LLMs. For inference across all AI models, see What is AI Inference?
Key takeaways
- Prefill processes input in parallel; decode generates tokens sequentially.
- The KV cache stores past attention results so each new token is cheaper to compute.
- Key metrics: time to first token (TTFT), tokens per second and throughput.
- Decode speed is usually limited by memory bandwidth, not raw compute.
- Batching, quantization, caching and speculative decoding reduce cost and latency.
LLM inference explained in simple terms
Imagine a writer asked to continue a story. First they read the whole story so far (prefill), which they can do quickly by skimming. Then they write the continuation word by word (decode), and each word needs a moment of thought. They keep notes about what they read, so they do not reread everything before each word. Those notes are the KV cache.
The two phases
| Phase | What happens | Performance characteristic |
|---|---|---|
| Prefill | All prompt tokens pass through the transformer in parallel; the KV cache is built | Compute-heavy; longer prompts increase time to first token |
| Decode | One new token per step, using the KV cache; repeat until done | Memory-bandwidth-heavy; sets tokens per second |
Key metrics
- Time to first token (TTFT): delay before output begins. Driven by prompt length and queueing.
- Tokens per second (per user): how fast text streams once started.
- Throughput: total tokens per second across all users on the hardware.
- Cost per million tokens: combines hardware cost and throughput.
A worked example: estimating response time
A model on a given server has a TTFT of 0.4 seconds for a 2,000-token prompt and generates 50 tokens per second. A 400-token answer takes about 0.4 + (400 ÷ 50) = 8.4 seconds. Cutting the answer to 150 tokens reduces it to about 3.4 seconds. This is why concise output instructions and structured output speed up applications, and why reasoning models, which generate many thinking tokens, are slower.
The KV cache
In attention, each token needs the keys and values of all earlier tokens. Recomputing them for every new token would be wasteful, so they are stored in the KV cache. The cache grows with context length and batch size and can consume large amounts of GPU memory. Techniques such as PagedAttention, introduced with the open-source vLLM engine in 2023, manage this memory more efficiently, allowing more concurrent users.
Techniques to speed up inference
| Technique | How it helps |
|---|---|
| Continuous batching | Groups many users’ requests to keep the GPU busy |
| Quantization | Smaller weights mean less memory to read per token |
| Prompt (prefix) caching | Reuses processing for repeated prompt prefixes, such as long system prompts |
| Speculative decoding | A small draft model proposes tokens; the large model verifies several at once |
| Distillation or smaller models | Less compute per token |
| Mixture of experts | Activates only part of the model per token |
| Streaming | Shows tokens as they are generated to improve perceived speed |
Where inference runs
- Hosted APIs: simplest; pay per token.
- Self-hosted cloud GPUs: serving engines such as vLLM or TensorRT-LLM for control and cost at scale.
- Local and edge: quantized open-weight models via tools like llama.cpp or Ollama.
Limitations
- Long contexts increase memory use and TTFT.
- Sequential decoding limits single-user speed.
- Costs scale with usage, especially output tokens.
- Performance varies widely between hardware and serving stacks.