Document chunking is the process of splitting documents into smaller passages so they can be embedded, stored and retrieved individually. Embedding models have input limits, context windows are finite, and retrieval works better on focused passages than whole documents. How you chunk determines what can be retrieved at all, which makes it one of the highest-impact decisions in a RAG pipeline.
Key takeaways
- Common chunk sizes are 200 to 800 tokens, with 10–20% overlap.
- Splitting on document structure beats splitting on character counts.
- Each chunk should make sense on its own.
- Adding surrounding context to each chunk improves retrieval significantly.
- Tables, code and lists usually need special handling.
Chunking explained in simple terms
Imagine cutting a manual into index cards for a librarian to hand out. Cut too finely and each card lacks context: “It must be replaced every six months” tells nobody what “it” is. Cut too coarsely and the card covers five topics, so retrieving it for one buries the other four. Good chunking makes each card a self-contained answer to some likely question.
Main strategies
| Strategy | How it splits | Best for |
|---|---|---|
| Fixed size | Every N tokens, with overlap | Simple, uniform text |
| Recursive | Try paragraphs, then sentences, then characters | General-purpose default |
| Structure-aware | By headings, sections or pages | Manuals, policies, documentation |
| Semantic | Where topic similarity drops between sentences | Unstructured prose |
| Element-based | Tables, code blocks and lists kept whole | Technical and financial documents |
| Sentence windows | Retrieve one sentence, return neighbours | Precise retrieval with context |
Choosing a chunk size
| Size | Effect |
|---|---|
| 100–200 tokens | Precise retrieval, often missing context |
| 300–500 tokens | Good general default |
| 600–1,000 tokens | More context, less precise matching |
| Whole document | Poor retrieval; everything matches weakly |
Smaller chunks match questions more sharply but risk losing meaning. Larger chunks carry context but dilute the embedding, since one vector must represent several ideas.
Overlap
Overlapping chunks by roughly 10–20% prevents answers being cut in half at a boundary. With 400-token chunks, a 50-token overlap means each chunk repeats the last 50 tokens of the previous one. The cost is a slightly larger index.
A worked example
An HR policy states: “Employees may carry forward unused leave. The maximum carry-forward is 15 days.“
- Chunked at 100 tokens with no overlap, the bolded sentence may land in a chunk that never mentions leave. A search for “leave carry forward limit” then misses it.
- Chunked by section with a heading prefix, the chunk begins “Leave Policy — Carry Forward” and includes both sentences, so it is retrieved reliably.
Adding context to chunks
A strong improvement is to prepend information to each chunk before embedding: the document title, the section heading, or a one-line summary of what the chunk is about within the document. This approach, sometimes called contextual retrieval, has been shown to cut retrieval failures substantially, because isolated chunks regain the context that made them meaningful.
Practical checklist
- Preserve headings and attach them to chunks.
- Keep tables and code blocks intact.
- Store metadata: source, title, section, date, permissions.
- Test retrieval with real questions after each chunking change.
- Re-chunk when you change embedding models or document formats.
Limitations
- No universally optimal size; it depends on content and questions.
- Re-chunking means re-embedding everything.
- Poorly parsed documents produce poor chunks regardless of strategy.
- Cross-chunk questions still need multi-passage retrieval.