Batch inference is the practice of processing several requests together in one pass rather than one at a time. Because generating a token requires reading the model’s weights from memory regardless of how many requests are being served, handling many at once uses the hardware far more efficiently. The term also covers offline batch processing, where large volumes of work are submitted for asynchronous handling at a discount.
Key takeaways
- Batching raises total tokens per second dramatically.
- Continuous batching adds and removes requests dynamically, avoiding idle slots.
- Larger batches can increase per-request latency.
- Batch size is limited by GPU memory, mainly the attention cache.
- Several providers offer asynchronous batch APIs at reduced prices, often around half the standard rate.
Batch inference explained in simple terms
A bus versus individual taxis. One bus trip serves 40 passengers for roughly the cost of one journey, though each passenger may wait a little longer and stop more often. For AI serving, the “journey” is reading the model weights, and batching lets many requests share it.
Static vs continuous batching
| Approach | How it works | Weakness |
|---|---|---|
| Static batching | Collect N requests, run them together until all finish | Short requests wait for the longest one, wasting capacity |
| Continuous batching | Add new requests as others finish, every step | More complex, but far higher utilisation |
Continuous batching, popularised by serving engines such as vLLM, is standard in modern LLM deployments. Combined with paged attention memory management, it allows many more concurrent users per GPU.
A worked example
A GPU serving one request at a time produces 60 tokens per second. With continuous batching at 32 concurrent requests, each stream might run at 40 tokens per second, but total throughput becomes 32 × 40 = 1,280 tokens per second. Throughput rises more than twentyfold while each user sees only a modest slowdown. Cost per million tokens falls in proportion.
Offline batch APIs
For work that does not need an immediate answer, several providers accept large jobs submitted as a file and returned within a window, commonly up to 24 hours, at a significant discount. This suits:
- Classifying or tagging large content archives.
- Generating thousands of product descriptions.
- Extracting fields from document backlogs.
- Creating embeddings for a whole knowledge base.
- Evaluating a model across a large test set.
What limits batch size
- GPU memory: the KV cache grows with batch size and context length.
- Latency targets: interactive apps cap how much queueing is acceptable.
- Sequence length variance: very long requests occupy memory for a long time.
Practical guidance
- Use continuous batching for interactive serving.
- Use offline batch APIs for bulk, latency-tolerant work.
- Monitor both throughput and p95 latency, not just averages.
- Group similar-length requests where possible for better efficiency.
Limitations
- Individual latency rises as batches grow.
- Memory constrains concurrency, especially with long contexts.
- Offline batch results are not available immediately.
- Failures in a batch job need careful retry handling.