Inference latency is the delay between sending a request to a model and receiving its response. For LLMs it is not a single number: users care about when output starts appearing (time to first token), how fast it continues (tokens per second) and when the answer is complete. Latency shapes user experience more than almost any other technical property.
Key takeaways
- Total latency = queueing + prompt processing (prefill) + token generation (decode) + network.
- Output length is usually the largest single factor.
- Latency and throughput trade off against each other.
- Streaming greatly improves perceived latency without changing total time.
- Reasoning models add latency because thinking tokens must be generated first.
Inference latency explained in simple terms
Ordering food at a busy counter, three things delay your meal: waiting in the queue, the kitchen reading your order, and the cooking itself. A model request works the same way, and complaining about “slow AI” usually means one of those three stages is the bottleneck.
The components
| Stage | What happens | Main driver |
|---|---|---|
| Queueing | Request waits for GPU capacity | Server load and batching policy |
| Prefill | The prompt is processed and the KV cache built | Prompt length |
| Decode | Tokens are generated one at a time | Output length and per-token speed |
| Network and overhead | Transport, authentication, serialisation | Distance and payload size |
A worked example
A request has a 3,000-token prompt and produces a 500-token answer on a system with 40 ms queueing, prefill of 300 ms and decode at 40 tokens per second.
- Queue: 0.04 s
- Prefill: 0.30 s
- Decode: 500 ÷ 40 = 12.5 s
- Network: 0.1 s
- Total: about 12.9 s, with first output visible after roughly 0.44 s.
Decode dominates. Halving the answer length saves over 6 seconds; halving the prompt saves about 0.15 seconds. This is why concise output instructions are the most effective latency fix in most applications.
Latency targets by use case
| Use case | Practical target |
|---|---|
| Voice agents | Under 500 ms to first audio |
| Interactive chat | Under 1 s to first token, steady streaming after |
| Coding autocomplete | Under 300 ms |
| Search summaries | 1–3 s total |
| Background or batch jobs | Seconds to minutes are acceptable |
How to reduce latency
- Shorten outputs: set maximum output tokens and ask for brevity.
- Shorten or cache prompts: prompt caching can cut prefill time sharply for repeated context.
- Use a smaller model for simple requests, routing hard ones to a larger one.
- Quantize to reduce memory traffic per token.
- Speculative decoding to produce several tokens per verification step.
- Stream so users see output immediately.
- Deploy closer to users to cut network time.
Latency versus throughput
Larger batch inference improves total tokens per second across all users but can make each individual request slower. Serving systems balance this with continuous batching. Interactive products optimise latency; bulk processing optimises throughput.
Limitations
- Measured latency varies with server load and time of day.
- Percentiles matter: p95 and p99 latency reveal problems averages hide.
- Provider latency can change without notice.
- Some latency, such as sequential decoding, is inherent to autoregressive generation.