Retrieval-augmented generation (RAG) is a technique in which relevant documents are retrieved from a knowledge source and added to an LLM’s prompt, so the model answers from that material rather than only from its training data. It is the standard way to make AI answer questions about company documents, product data or anything newer than the model’s knowledge cut-off. The approach was named in a 2020 paper by Patrick Lewis and colleagues.
Key takeaways
- RAG adds retrieved context to the prompt; it does not change the model’s weights.
- It reduces hallucination by grounding answers in real sources.
- It handles changing information far better than fine-tuning.
- It makes citations possible, so users can verify answers.
- Retrieval quality sets the ceiling: if the right passage is not retrieved, the answer will be wrong.
RAG explained in simple terms
Closed-book exams test memory. Open-book exams let the student look things up. A plain LLM answers from memory, which is broad but fixed and fuzzy. RAG hands it the right pages before it answers, so it can quote the current policy rather than recalling something similar from training.
How RAG works
- Index: documents are split into chunks, converted to vector embeddings and stored in a vector database.
- Retrieve: the user’s question is embedded and the most relevant chunks are found.
- Augment: those chunks are inserted into the prompt with instructions to answer only from them.
- Generate: the model writes the answer, ideally citing the sources used.
The full engineering version of this is covered in What is a RAG pipeline?
A worked example
An employee asks an internal assistant, “How many days of paternity leave do I get?”
- Without RAG: the model answers from general knowledge, perhaps quoting a common policy from elsewhere. It sounds confident and may be wrong.
- With RAG: the system retrieves the HR policy chunk stating 15 working days within six months of birth, and the model answers with that figure and a link to the policy page.
RAG vs fine-tuning vs long context
| Approach | Best for | Weakness |
|---|---|---|
| RAG | Current, private or frequently changing facts | Retrieval must find the right content |
| Fine-tuning | Consistent format, tone, narrow tasks | Poor at teaching changing facts |
| Long context | Small document sets you can paste wholesale | Expensive, slower, and quality drops on very long inputs |
These combine well: a fine-tuned model for style, RAG for facts, within a sensible context budget.
Why teams choose RAG
- Answers reflect today’s documents, not last year’s training data.
- Private data stays in your systems and is only sent as needed.
- Sources can be cited and audited.
- Updating knowledge means re-indexing a document, not retraining a model.
- Access control can be applied at retrieval time, so users only see what they are allowed to.
Common failure points
- The answer exists but is not retrieved, often due to poor chunking or a weak retriever.
- Retrieved chunks are related but do not answer the question.
- The model ignores the context and answers from memory anyway.
- Conflicting documents produce confused answers.
- Stale content is never removed from the index.
Limitations
- Adds latency and complexity compared with a plain model call.
- Quality depends on document quality and structure.
- It reduces but does not eliminate hallucination.
- Multi-step questions may need several retrieval rounds.