Time to first token (TTFT) is the delay between sending a request and receiving the model’s first output token. It covers queueing and prefill, the stage where the model processes the entire prompt and builds its attention cache. Because it decides how long a user stares at a blank screen, TTFT is the latency metric that most affects perceived responsiveness.
Key takeaways
- TTFT ≈ queue time + prefill time + network overhead.
- Prompt length is the main controllable driver.
- Prompt caching can reduce TTFT dramatically for repeated context.
- Typical interactive targets are under 1 second, and under 500 ms for voice.
- After the first token, tokens per second takes over as the relevant metric.
TTFT explained in simple terms
You ask a colleague a question. The pause before they begin speaking is TTFT. They may need to read your long email first (prefill) or finish what they were doing (queueing). Once they start talking, a different measure applies: how quickly they speak.
What drives TTFT
| Factor | Effect |
|---|---|
| Prompt length | Longer prompts need more prefill compute |
| Server load | Busy servers queue requests |
| Model size | Larger models process prompts more slowly |
| Cold start | Loading a model into GPU memory adds seconds or more |
| Network distance | Cross-region requests add round-trip time |
| Prompt caching | Reused prefixes skip most of the prefill work |
A worked example
An assistant sends a 200-token system prompt, 8,000 tokens of retrieved documents and a 50-token question.
- Without caching: the full 8,250 tokens are processed at prefill, giving a TTFT of, say, 900 ms.
- With prompt caching of the system prompt and documents: only the new question is processed, and TTFT drops to perhaps 200 ms.
Since retrieval-heavy applications resend large context repeatedly, caching is often the single biggest TTFT improvement available.
Typical targets
| Application | TTFT target |
|---|---|
| Voice agents | Under 500 ms |
| Chat interfaces | Under 1 s |
| Code completion | 100–300 ms |
| Document analysis | 1–3 s acceptable |
| Batch jobs | Not critical |
How to improve TTFT
- Cache prompts: keep stable content, such as system prompts and manuals, at the start so it can be reused.
- Trim context: retrieve fewer, more relevant chunks instead of large documents.
- Use a smaller model for simple requests, with routing for harder ones.
- Keep servers warm to avoid cold starts.
- Deploy near users to reduce network time.
- Show immediate feedback in the interface while prefill runs.
TTFT and reasoning models
Reasoning models may think for many tokens before producing visible output. If the interface hides thinking, effective TTFT can be many seconds. Showing a thinking indicator, or streaming a summary of progress, keeps the experience acceptable.
Limitations
- A good average TTFT can hide poor p95 and p99 values under load.
- Prefill scales with prompt length, so long-context features cost responsiveness.
- Cold starts are hard to avoid entirely in serverless deployments.
- Improving TTFT does not make the full response finish sooner.