A reranker is a model that reorders retrieved candidates by scoring each one directly against the query. Initial retrieval is fast but approximate: it compares pre-computed embeddings that were created without knowing the query. A reranker reads the query and each passage together, which is far more accurate. The standard pattern is to retrieve many candidates cheaply, then rerank and keep the best few.
Key takeaways
- Retrieval uses a bi-encoder; reranking uses a cross-encoder.
- Cross-encoders are much more accurate and much slower, so they run on a shortlist.
- A typical setup retrieves 50 candidates and reranks down to 3–5.
- Adding a reranker is often the single largest quality improvement in RAG.
- It costs extra latency, usually tens to a couple of hundred milliseconds.
A reranker explained in simple terms
A recruiter screens 500 CVs with keyword filters to get 50, then actually reads those 50 before shortlisting 5. Screening is fast and rough; reading is slow and accurate. Retrieval is the screen, reranking is the read.
Bi-encoder vs cross-encoder
| Aspect | Bi-encoder (retrieval) | Cross-encoder (reranking) |
|---|---|---|
| How it works | Embeds query and documents separately | Processes query and document together |
| Precomputation | Documents embedded once, in advance | Nothing can be precomputed |
| Speed | Milliseconds across millions | Tens of milliseconds per document |
| Accuracy | Good | Substantially better |
| Scale | Whole corpus | A shortlist only |
Because a cross-encoder sees both texts at once, it can judge whether a passage actually answers the question, not merely whether it is about the same topic.
The retrieve-then-rerank pattern
- Retrieve 30–100 candidates with vector or hybrid search.
- Send each candidate with the query to the reranker.
- Receive a relevance score per candidate.
- Keep the top 3–5 and build the prompt from them.
A worked example
Question: “Can I claim HRA if I live with my parents?”
Vector search returns 50 chunks. Several discuss house rent allowance generally; one covers rent paid to parents specifically, but it ranks eleventh because its wording differs from the query. A reranker reads each chunk against the question, recognises that the eleventh chunk directly answers it, and moves it to first place. The model now receives the right passage instead of three vaguely related ones.
Types of rerankers
| Type | Notes |
|---|---|
| Hosted reranking APIs | Managed cross-encoder services, simple to adopt |
| Open cross-encoder models | Self-hosted models such as BGE rerankers |
| LLM-as-reranker | Prompt an LLM to score relevance; flexible but slower and costlier |
| Multilingual rerankers | Needed when queries and documents span languages |
When a reranker helps most
- Large corpora where many passages look superficially similar.
- Questions whose wording differs from the source text.
- Systems limited to a few context chunks.
- Technical content where small distinctions matter.
- Any RAG pipeline where retrieval recall is good but precision is poor.
Costs and tuning
- Latency scales with the number of candidates reranked.
- Start with 50 candidates and measure whether more helps.
- Hosted rerankers charge per document scored.
- Cache results for repeated queries where appropriate.
Limitations
- Cannot rescue candidates that retrieval never returned.
- Adds latency and cost per query.
- Another model to host, monitor and update.
- Quality varies across languages and domains.