Area 02 // RAGLesson 38 / 14013 min read
Reranking
Reranking is a second, more careful pass over the candidates your search returned, using a model that reads the question and each candidate together, so the truly most relevant chunks end up at the top before you send them to the LLM.
Think of it like this
Analogy
A job opening gets 500 applications. A recruiter skims them all in an hour using keywords and gut feeling, and keeps 30. Then the hiring manager reads those 30 carefully, comparing each against the actual job, and ranks them. Only the top 5 get interviews.
Fast retrieval is the recruiter's skim. The reranker is the hiring manager's careful read.
The problem it solves
Vector and keyword search are built for speed: they compare precomputed representations, so they can't look closely at how a specific chunk answers a specific question. Their top 10 usually contains the right chunks, but not always in the right order: the best chunk might be 7th, behind near-misses that share words or a topic.
You can only send a few chunks to the LLM (cost, latency, lost in the middle). If the best chunk sits at position 7 and you send 5, the answer is wrong even though retrieval "found" it.
How it works, step by step
- First stage: retrieve wide. Use fast search (vector, BM25, hybrid) to get 20–100 candidates. Aim for high recall: the right chunk should be somewhere in this list.
- Understand the two model types:
- A bi-encoder (an embedding model) encodes the query and each document separately. Document vectors are precomputed, so search is fast, but the model never sees them side by side.
- A cross-encoder reads the query and one document together in a single pass, with attention between their words, and outputs a relevance score. Much more precise, but it must run for every pair at query time.
- Second stage: rerank. Score each (query, candidate) pair with the reranker and sort by that score.
- Keep the top few (3–10) for the prompt, optionally dropping those below a tuned score threshold.
- Choose a reranker (as of November 2026): hosted rerank APIs (Cohere Rerank, Voyage, Jina), open cross-encoders (BGE rerankers, ms-marco MiniLM models) you can run on CPU or GPU, or an LLM asked to score or order candidates (flexible, slower, pricier).
- Measure ranking quality (MRR, nDCG) and answer quality with and without reranking, plus the added latency.
See it in code
- Needs
- Nothing for the offline version (plain Python 3). The real rerankers below need sentence-transformers (local) or a rerank API key.
A stand-in for a cross-encoder (a simple scoring function that reads query and document together) shows the effect: the best answer moves from 2nd to 1st.
query = "can I return a laptop I already opened"
# Stage 1: fast retrieval returned these 4 candidates (order from vector search).
candidates = [
"Laptops come with a 2-year warranty.",
"Opened laptops can be returned within 14 days if undamaged.",
"You can return unopened items within 30 days.",
"Opened software cannot be returned.",
]
def toy_cross_encoder(q: str, doc: str) -> float:
"""Stand-in for a real reranker model, which reads query AND document together.
Here: reward documents that contain the query's key words, extra for 'opened' + 'laptop'."""
words = {"return", "laptop", "opened"}
d = doc.lower()
score = sum(w in d for w in words)
return score + (1.5 if "opened laptop" in d else 0)
# Stage 2: score each (query, document) pair and re-sort.
reranked = sorted(candidates, key=lambda d: -toy_cross_encoder(query, d))
print("before:", candidates[0])
print("after :", reranked[0])
print("keep top 2 for the prompt:", reranked[:2])before: Laptops come with a 2-year warranty.
after : Opened laptops can be returned within 14 days if undamaged.
keep top 2 for the prompt: ['Opened laptops can be returned within 14 days if undamaged.', 'You can return unopened items within 30 days.']With a real open cross-encoder (runs locally, no key) it's three lines:
from sentence_transformers import CrossEncoder # pip install sentence-transformers
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2") # small, CPU-friendly
scores = model.predict([(query, doc) for doc in candidates]) # one score per pair
reranked = [doc for _, doc in sorted(zip(scores, candidates), reverse=True)]A real example
Example
Legal search: the right clause was always there, just too low
A contracts team's RAG system retrieved the top 5 chunks by vector similarity. In their eval set, the correct clause was in the top 20 for 92% of questions, but in the top 5 for only 68%. Answers were wrong a third of the time even though search "worked".
They changed to: hybrid search top 40 → cross-encoder rerank → top 6. The correct clause reached the top 6 in 89% of questions, and answer accuracy jumped accordingly. Reranking 40 candidates added about 150 ms on a small GPU, which they judged well worth it.
The easy way
Start here
Retrieve 20–50 candidates instead of 5, rerank with a small open cross-encoder or a hosted rerank API, and keep the top 5. Measure your top-5 hit rate before and after on 30 questions.
When you grow: the production setup
Production systems cap candidates to bound latency, batch reranker calls, run rerankers on GPUs or use hosted APIs with timeouts and fallbacks (skip reranking if it's slow or down), cache results for repeated queries, fine-tune rerankers on their own click or label data, and track the reranker's contribution with ablation evals.
Trade-offs
What you gain
- Often the single biggest quality boost for a RAG system after basic retrieval.
- Lets the first stage favour recall, and sends fewer, better chunks to the LLM.
- Works on top of any retriever.
What it costs you
- Adds latency for every candidate scored.
- Another model to host or another API to pay for.
- Can't fix candidates that retrieval never found.
When not to use it
- First-stage retrieval already puts the right chunk at the top almost always (measure it).
- Extremely latency-sensitive paths where even 100 ms matters.
Common mistakes
- Retrieving only 5 candidates and then reranking, so there's nothing to improve.
- Reranking hundreds of candidates and blowing the latency budget.
- Expecting reranking to fix missing documents (that's a retrieval/recall problem).
- Not measuring with and without the reranker.
Interview questions
Try answering out loud first, then open the model answer.
What's the difference between a bi-encoder and a cross-encoder?
Model answer
A bi-encoder embeds query and document independently into vectors and scores them with a similarity function; document vectors can be precomputed and indexed, so it scales to millions with ANN search, but it can't model fine-grained interactions. A cross-encoder concatenates query and document and runs them through the model jointly, producing a relevance score with full attention between them: far more accurate, but computed per pair at query time, so it's used to rerank a short candidate list.
How do you design a two-stage retrieval system?
Model answer
Stage one maximises recall cheaply: hybrid BM25 plus dense retrieval returning the top 20–100 with metadata filters. Stage two maximises precision: a cross-encoder or LLM reranker scores each candidate with the query, and the top 3–10 go to the prompt, possibly with a score threshold. Tune candidate counts against latency budgets and measure Recall@k for stage one and MRR/nDCG for stage two.
When might you use an LLM as a reranker?
Model answer
When you need nuanced relevance judgements (complex questions, domain reasoning, multi-condition criteria) and can afford the latency and cost, or when no good reranker exists for your language or domain. Use a small, fast model, score candidates in batches with structured outputs, and compare against a cross-encoder on your eval set.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Reranker
- A model that re-scores a short list of retrieved candidates more carefully, so the most relevant ones end up first.
- Cross-encoder
- A model that reads a query and a document together and outputs a relevance score. Precise but too slow to run on a whole corpus.
- Bi-encoder
- A model that embeds queries and documents separately into vectors, so documents can be indexed ahead of time. Fast, less precise.
- Retrieval
- Finding the pieces of stored information most relevant to a question, for example by keyword or vector search.
- Recall
- The share of the truly relevant items that a search (or classifier) actually finds.
Before and after
Progress is saved in this browser only.