A RAG pipeline is the end-to-end system that turns raw documents into grounded answers. It has two halves: an ingestion pipeline that prepares and indexes content, usually run in the background, and a query pipeline that runs on every user question. RAG is the idea; the pipeline is the engineering.
Key takeaways
- Ingestion runs offline; retrieval and generation run per request.
- Most RAG quality problems come from ingestion, not the model.
- Adding a reranker is one of the highest-return improvements.
- Every stage should be measured separately, not just the final answer.
- Access control and freshness must be designed in from the start.
The ingestion pipeline
| Stage | What happens | Common pitfalls |
|---|---|---|
| Collect | Pull documents from drives, wikis, CMS, databases | Missing sources, no permissions model |
| Parse | Extract text from PDF, DOCX, HTML, slides | Broken tables, lost headings, OCR errors |
| Clean | Remove boilerplate, navigation, duplicates | Deleting useful context |
| Chunk | Split into retrievable passages | Chunks that cut across a single idea |
| Enrich | Add titles, dates, source URLs, access tags as metadata | No metadata, so filtering is impossible |
| Embed | Convert chunks to vectors | Mixing embedding models |
| Store | Write to a vector database with metadata | No update or delete strategy |
The query pipeline
- Receive the user question and conversation history.
- Rewrite the query if needed, resolving references like “it” from earlier turns.
- Retrieve candidates with vector search, keyword search or hybrid search.
- Filter by metadata, including the user’s access rights.
- Rerank candidates with a cross-encoder and keep the best few.
- Assemble the prompt with instructions, context and citation rules.
- Generate the answer with grounding instructions.
- Post-process: attach citations, verify claims, log everything.
A worked example
A support bot receives: “Can I still return it after 20 days?”
- Query rewriting turns this into “return policy time limit for wireless headphones”, using the earlier turns.
- Hybrid retrieval returns 50 candidate chunks.
- Metadata filtering removes chunks from an outdated policy version.
- A reranker scores them and keeps the top 4.
- The prompt instructs the model to answer only from those chunks and cite them.
- The answer quotes the 30-day window and links to the policy page.
Evaluating a RAG pipeline
Measure retrieval and generation separately, or you will not know which one failed.
| Stage | Metric |
|---|---|
| Retrieval | Recall@k: is the correct passage in the retrieved set? |
| Reranking | Precision@k, mean reciprocal rank |
| Generation | Faithfulness: is every claim supported by the context? |
| Generation | Answer relevance: does it address the question? |
| End to end | Human review on a fixed question set |
Build a test set of 50 to 200 real questions with known correct sources, and re-run it after every change.
Operational concerns
- Freshness: re-index changed documents and delete removed ones.
- Permissions: filter at retrieval time so users cannot see restricted content.
- Cost: embedding is a one-off per chunk; retrieval and generation cost per query.
- Latency: retrieval and reranking add to time to first token.
- Monitoring: log queries with no good match, which reveal content gaps.
Limitations
- Many moving parts, each a potential failure point.
- Poor source documents cannot be fixed downstream.
- Multi-hop questions may need iterative retrieval.
- Pipelines need ongoing maintenance as content changes.