Area 02 // RAGLesson 45 / 14015 min read

Measuring retrieval: Recall@k, MRR and nDCG

Retrieval metrics tell you, for a set of test questions with known relevant documents, how often the right documents are found (Recall@k), how high the first right one appears (MRR), and how good the whole ranking is (nDCG), so you can improve search with numbers instead of guesses.

On this page

Think of it like this
The problem it solves
How it works: Recall@k
How it works: MRR and nDCG
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

You ask a librarian for books about volcanoes, and they hand you a stack of 10.

Recall@10: of all the library's volcano books, how many are in your stack?
MRR: how far down the stack is the first volcano book? (Top = great, 8th = poor.)
nDCG: are the best volcano books near the top of the stack, or buried under the so-so ones?

The problem it solves

Teams tune chunk sizes, embedding models, hybrid weights and rerankers by trying a few questions by hand and going with whatever "looks better". That's how changes that help five demo questions quietly hurt fifty others. And because most RAG failures are retrieval failures, you need a precise, repeatable measurement of retrieval on its own, separate from how the LLM writes.

How it works: Recall@k

Build a test set: 50–500 real questions, each labelled with the IDs of the chunks or documents that answer it (a golden dataset).
Run retrieval for each question and keep the ranked list of returned IDs.
Recall@k for one question = (relevant items found in the top k) ÷ (total relevant items).
Average over all questions.
Pick k to match your system: if you send 5 chunks to the LLM, Recall@5 is what matters; if you rerank the top 50, Recall@50 measures the first stage.
Related: Precision@k = relevant items in the top k ÷ k. Useful when you care about not wasting prompt space on irrelevant chunks.

How it works: MRR and nDCG

Reciprocal rank for one question = 1 ÷ position of the first relevant result (1st → 1.0, 2nd → 0.5, 4th → 0.25, none → 0).
MRR (mean reciprocal rank) = the average over questions. Great when one good result is enough (FAQ lookup, navigation).
DCG@k adds up relevance scores of the top k results, each divided by log₂(position + 1), so lower positions count less.
Ideal DCG is the DCG of the perfect ordering.
nDCG@k = DCG ÷ ideal DCG, between 0 and 1. It considers every relevant result and their order, and supports graded relevance (2 = perfect, 1 = partly relevant). Best for judging rerankers and full rankings.
Report several together and compare variants on the same test set; small differences on small sets may be noise (confidence intervals).

Metric	Answers the question	Use it for
Recall@k	Did we find the right stuff in the top k?	First-stage retrieval, choosing k
Precision@k	How much of the top k is useful?	Prompt efficiency
MRR	How soon is the first right result?	Single-answer lookups
nDCG@k	Is the whole ranking good?	Rerankers, graded relevance

See it in code

Needs: Nothing: plain Python 3.

All three metrics on three test questions. Question 3's relevant document was never retrieved, which drags every metric down.

retrieval_metrics.py

import math
 
# For each test question: the IDs the retriever returned (in order) and the IDs that are truly relevant.
tests = [
    {"retrieved": ["d3", "d7", "d1", "d9", "d2"], "relevant": {"d1", "d2"}},
    {"retrieved": ["d4", "d5", "d6", "d8", "d0"], "relevant": {"d5"}},
    {"retrieved": ["d2", "d1", "d3", "d4", "d5"], "relevant": {"d9"}},
]
 
def recall_at_k(t, k):        # share of relevant docs found in the top k
    return len(set(t["retrieved"][:k]) & t["relevant"]) / len(t["relevant"])
 
def reciprocal_rank(t):       # 1 / position of the FIRST relevant doc (0 if none)
    return next((1 / (i + 1) for i, d in enumerate(t["retrieved"]) if d in t["relevant"]), 0.0)
 
def ndcg_at_k(t, k):          # rewards relevant docs more the higher they are ranked
    dcg = sum(1 / math.log2(i + 2) for i, d in enumerate(t["retrieved"][:k]) if d in t["relevant"])
    ideal = sum(1 / math.log2(i + 2) for i in range(min(k, len(t["relevant"]))))
    return dcg / ideal
 
for k in (3, 5):
    print(f"Recall@{k}: {sum(recall_at_k(t, k) for t in tests) / len(tests):.2f}")
print(f"MRR     : {sum(reciprocal_rank(t) for t in tests) / len(tests):.2f}")
print(f"nDCG@5  : {sum(ndcg_at_k(t, 5) for t in tests) / len(tests):.2f}")

Output

Recall@3: 0.50
Recall@5: 0.67
MRR     : 0.28
nDCG@5  : 0.39

Work it through by hand for question 1: d1 is at position 3 and d2 at 5, so Recall@3 = 1/2, Recall@5 = 2/2, reciprocal rank = 1/3. Libraries like Ragas, ranx and pytrec_eval compute these at scale.

A real example

Example

Choosing between three retrieval setups with numbers

A support team argued for weeks about chunk sizes and whether hybrid search was worth it. They labelled 250 real customer questions with the help-centre chunks that answer them (two people, three days), then ran:

Setup	Recall@5	MRR	nDCG@5
Vector, 1,000-char chunks	0.62	0.48	0.51
Vector, paragraph chunks + titles	0.71	0.57	0.60
Hybrid + reranker, paragraph chunks	0.86	0.74	0.77

The debate ended in an afternoon. They added the evaluation to CI so every future change to retrieval is measured the same way (regression evals).

The easy way

Start here

Write down 30 real questions and, for each, the chunk ID(s) that answer it. Compute Recall@5 for your current setup with the snippet above. Every time you change chunking, embeddings or search, re-run it. That single number will guide most of your RAG decisions.

When you grow: the production setup

Teams maintain versioned retrieval test sets per domain (grown from production failures), use graded relevance labels, compute metrics with confidence intervals, slice results by query type (codes, follow-ups, multilingual), track them in CI and dashboards, and pair them with end-to-end metrics like faithfulness and context precision.

Trade-offs

What you gain

Objective, repeatable numbers for every retrieval change.
Isolates retrieval problems from generation problems.
Cheap to compute once the test set exists.

What it costs you

Needs labelled questions, which take human time to create and maintain.
Labels can be incomplete (unlabelled relevant chunks get counted as misses).
Good retrieval metrics don't guarantee good final answers.

When not to use it

Never skip measurement entirely; but for a throwaway prototype, 20 hand-checked questions are enough to start.

Common mistakes

Tuning retrieval by eyeballing a few demo questions.
Using Recall@50 when only 5 chunks reach the prompt.
Labels tied to chunk IDs that change when you re-chunk (label documents or text spans instead, then map).
Reading tiny metric changes on small test sets as real improvements.
Measuring only final answers and never retrieval on its own.

Interview questions

Try answering out loud first, then open the model answer.

Explain Recall@k, MRR and nDCG and when you'd use each.

Model answer

Recall@k is the fraction of relevant items retrieved in the top k: use it for first-stage retrieval and choosing k. MRR averages the reciprocal of the first relevant result's rank: use it when one correct result suffices. nDCG@k sums graded relevance discounted by log of position, normalised by the ideal ordering: use it for full ranking quality and rerankers. I'd report them together on a fixed labelled set.

How do you build a test set for retrieval evaluation?

Model answer

Sample real user questions (and add important edge cases), have domain experts label the relevant documents or passages, ideally with graded relevance, keep labels at a level that survives re-chunking (document or span), include unanswerable questions, version the set, and keep growing it from production failures. LLM-generated questions from documents can bootstrap it but should be reviewed.

Your Recall@5 is high but answers are still poor. What's going on?

Model answer

Retrieval is finding the right chunks, so look at generation: the prompt may not enforce grounding, too many irrelevant chunks may distract the model, the right chunk may sit in the middle of a long context, chunks may be truncated, or the model is too weak for the reasoning required. Measure faithfulness and answer relevance, inspect failing cases, and adjust prompt, context ordering or model.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Recall@k: The share of relevant items that appear in the top k search results, averaged over test questions.
MRR (mean reciprocal rank): The average over test questions of 1 divided by the position of the first relevant result.
nDCG: Normalised discounted cumulative gain. Scores a whole ranking, rewarding relevant results more the higher they appear.
Precision@k: The share of the top k search results that are relevant.
Golden dataset: A trusted, versioned set of test inputs with expected outputs or labels, used to measure an AI system repeatably.

Before and after

Read first