A context window is the maximum amount of text, measured in tokens, that a large language model can consider at one time. Everything the model uses to answer must fit inside it: the system prompt, your message, earlier conversation turns, any pasted or retrieved documents, and the response being generated. Anything outside the window is invisible to the model.
Key takeaways
- The context window is the model’s working memory for one request.
- It includes both input and output tokens.
- Context sizes have grown from about 2,000 tokens in GPT-3 (2020) to hundreds of thousands, and in some models over a million.
- A bigger window does not guarantee the model uses all information equally well.
- Longer contexts cost more and respond more slowly.
A context window explained in simple terms
Imagine a desk. You can only work with the papers that fit on the desk right now. A small desk holds a few pages; a large desk holds several books. If you add more papers, the oldest fall off the edge. An LLM’s context window is its desk. The model does not remember past conversations unless they are placed back on the desk.
What fills the context window
| Item | Example |
|---|---|
| System prompt | Instructions, rules and persona |
| Conversation history | Earlier user and assistant messages |
| Retrieved documents | Help articles or files added for grounding |
| Tool results | Search results, database rows, API responses |
| The current prompt | The user’s latest question |
| The response | Output tokens being generated |
A worked example: fitting a contract
Suppose a model has a 128,000-token context window. You want to analyse a 150-page contract of about 75,000 tokens, with a 1,500-token system prompt, and allow up to 4,000 tokens for the answer.
Total: 75,000 + 1,500 + 4,000 = 80,500 tokens, which fits. Adding three more similar contracts would exceed the window, so you would need to summarise them first or retrieve only the relevant sections.
Why bigger is not always better
- “Lost in the middle”: a 2023 study by Nelson Liu and colleagues found models often used information at the start and end of long inputs better than information in the middle.
- Cost: you pay for every input token on every request.
- Speed: longer inputs take longer to process before the first output token appears.
- Noise: irrelevant text can distract the model and reduce accuracy.
Context window vs memory
The context window is temporary. Once a conversation ends, or older messages are dropped to save space, the model no longer has that information. Some applications add “memory” features that store facts separately and insert relevant ones into future context windows. The model’s permanent knowledge comes from training, not the context window.
How applications manage limited context
- Truncation: drop the oldest messages.
- Summarisation: replace old conversation turns with a short summary.
- Retrieval (RAG): store documents outside the model and insert only the most relevant chunks.
- Prompt caching: some providers reuse processed prompt prefixes to cut cost and latency on repeated context.
- Chunking: process long documents section by section, then combine results.
Why context windows have limits
In a standard transformer, attention compares every token with every other token, so compute grows roughly with the square of context length. The model also stores a key-value cache for every token during inference, which consumes GPU memory. Research into efficient attention and memory techniques has steadily pushed limits higher.
Best practices
- Put the most important instructions and information clearly, near the start or end.
- Include only relevant documents rather than everything available.
- Ask the model to quote or cite the parts of the context it used.
- Test performance on long inputs before relying on the full window.