Tokens per second (TPS) measures how quickly a model generates output tokens. It is reported in two different ways that are often confused: per-user speed, which determines how fast text appears on one person’s screen, and total throughput, which counts all tokens produced across every concurrent request on the hardware. Optimising one often comes at the expense of the other.
Key takeaways
- Per-user TPS governs the streaming experience; throughput governs cost per token.
- Decode speed is usually limited by memory bandwidth, not raw compute.
- Roughly 10–20 tokens per second exceeds normal human reading speed.
- Larger batches raise throughput but can lower per-user speed.
- Quantization and speculative decoding improve both in many setups.
Tokens per second explained in simple terms
Per-user TPS is how fast one cashier serves the customer in front of them. Throughput is how many customers the whole shop serves per hour. Adding more lanes per cashier raises the shop’s total but may slow each individual transaction slightly.
Why decode speed is memory-bound
To produce one token, the system must read the model’s weights from GPU memory. With a 7-billion-parameter model at 16-bit precision, that is about 14 GB of data read per token, per request stream. Modern GPUs have very high memory bandwidth, but this read is the bottleneck, not arithmetic. Two consequences follow:
- Smaller or quantized models generate faster, because less data is read.
- Batching many requests together is nearly free in bandwidth terms, since the same weights serve every request in the batch. This is why throughput scales well with batch inference.
A worked example
A 500-token answer at different per-user speeds:
| Speed | Time to complete |
|---|---|
| 10 tokens/s | 50 s |
| 30 tokens/s | 17 s |
| 60 tokens/s | 8 s |
| 120 tokens/s | 4 s |
Above roughly 20 tokens per second, most users are reading rather than waiting, so further gains matter mainly for agents and automated pipelines that consume the whole output at once.
Per-user speed vs throughput
| Metric | Measures | Improved by | Matters for |
|---|---|---|---|
| Per-user TPS | One stream’s speed | Smaller models, quantization, speculative decoding | User experience |
| Total throughput | All tokens across requests | Continuous batching, larger batches, efficient serving | Cost per million tokens |
How to improve tokens per second
- Use a smaller or distilled model where quality allows.
- Quantize weights to reduce memory traffic.
- Apply speculative decoding to produce several tokens per verification step.
- Use an optimised serving engine with continuous batching and efficient attention.
- Keep outputs short, which reduces total time even at the same speed.
- Choose hardware with higher memory bandwidth.
Limitations
- Published TPS figures depend heavily on hardware, batch size and context length.
- Speed drops as context grows, because the attention cache gets larger to read.
- Fast generation of poor answers is not a win.
- Tokenizer differences mean token counts, and therefore TPS, are not comparable across models.