Maximum output tokens (often `max_tokens` or `max_output_tokens`) is the setting that caps how many tokens a model may generate in a single response. When the cap is reached, generation stops immediately, even mid-sentence. It is one of the simplest and most effective controls over cost, latency and response length.
Key takeaways
- It limits output only; the prompt is counted separately as input.
- Output tokens plus input tokens must fit within the context window.
- Hitting the cap truncates the answer; APIs report a finish reason such as “length”.
- Output tokens are usually the most expensive part of a request.
- Setting it high does not make answers longer; it only permits longer answers.
Maximum output tokens explained in simple terms
Think of giving a speaker a strict time limit. They might finish early, which is fine. But if they are still talking when the buzzer sounds, they are cut off mid-word. The limit does not make them speak longer; it only caps how long they may go.
A worked example
A model has a 128,000-token context window. Your request uses:
- System prompt: 800 tokens
- Retrieved documents: 6,000 tokens
- User question: 200 tokens
- Total input: 7,000 tokens
If you set maximum output tokens to 2,000, the request needs 9,000 tokens of the window in total, which fits comfortably. If you set it to 125,000, most providers will reject the request because input plus maximum output exceeds the window.
What it controls
| Effect | Why |
|---|---|
| Cost ceiling | Output tokens are billed, often at a higher rate than input |
| Worst-case latency | Generation time scales with the number of tokens produced |
| Truncation risk | Too low a value cuts responses mid-sentence |
| Memory reservation | Some serving systems reserve KV cache space based on the limit |
How to choose a value
- Estimate the longest reasonable answer for your task.
- Add headroom of roughly 20–50%.
- Test with real inputs and check for truncated responses.
- For structured output, make sure the full JSON can fit, since truncated JSON fails to parse.
Rough guides: a short chat reply 200–500 tokens, a detailed explanation 800–1,500, a long document section 2,000–4,000. Reasoning models need far more, because thinking tokens usually count towards the output budget.
Limit versus instruction
The setting is a hard stop, not a style instruction. To get genuinely shorter answers, ask for them in the prompt (“answer in under 100 words”) and keep the token cap as a safety net. Relying on the cap alone produces answers that simply stop mid-thought.
Handling truncation
- Check the finish reason returned by the API.
- Ask the model to continue from where it stopped.
- Break long tasks into sections.
- Request more concise output rather than raising the cap indefinitely.
Limitations
- Cannot guarantee a complete, well-formed answer.
- Different providers use different parameter names and defaults.
- Models vary in how well they respect length instructions.
- Very high limits can reduce how many requests a server handles concurrently.