Area 04 // EvalsLesson 71 / 14015 min read
RAG evals: faithfulness, relevance and context precision
RAG systems are evaluated by checking the answer is faithful to the retrieved sources (no invented claims), relevant to the question (answers what was asked), and that retrieval put useful context at the top (context precision), so you can tell whether a bad answer came from retrieval or generation.
Think of it like this
Analogy
Grading a student's open-book exam answer:
- Faithfulness: is everything they wrote actually in the book? (No made-up facts.)
- Answer relevance: did they answer the question that was asked?
- Context precision: did they open the right pages, rather than flipping through irrelevant chapters first?
A student can fail any one of these independently, and each points to a different fix.
The problem it solves
A RAG answer can go wrong in several places: retrieval misses the right chunk, retrieval returns junk, the model ignores the context, the model invents facts, or the answer wanders off the question. A single "quality" score can't tell you which. You need separate metrics for retrieval and generation so you fix the right component.
How it works: faithfulness
- Split the answer into claims (an LLM does this): "You get 25 days of leave." "You can carry over 5 days." "Unused days are paid out in December."
- Check each claim against the retrieved context (an LLM judge or NLI model): supported or not?
- Faithfulness = supported claims ÷ total claims. 1.0 means every claim is grounded.
- Low faithfulness means hallucination: fix the prompt (answer only from sources, cite, abstain), the model, or add citation checks.
How it works: answer relevance
- Ask whether the answer addresses the question, and every part of it, without padding or drifting.
- Methods: an LLM judge with a rubric ("answers all parts", "no unrelated content"), or generating questions from the answer and comparing them with the original question (embedding similarity), as some tools do.
- Answer relevance catches answers that are grounded but miss the point, half-answer compound questions, or bury the answer in boilerplate.
How it works: context precision
- Label each retrieved chunk as relevant or not for the question (by humans, or an LLM judge compared against a reference answer).
- Context precision rewards relevant chunks ranked near the top: compute precision at each rank where a relevant chunk appears and average them (like average precision).
- Pair it with context recall: did the retrieved context contain everything needed (checked against a reference answer or labelled relevant chunks)?
- Low precision → too much noise in the prompt (tune k, add reranking). Low recall → retrieval misses information (chunking, embeddings, hybrid search, query rewriting).
| Symptom | Metric that shows it | Fix in |
|---|---|---|
| Invented facts | Low faithfulness | Prompt, model, citation checks |
| Off-topic or partial answers | Low answer relevance | Prompt, model, query understanding |
| Noisy context | Low context precision | k, reranking, filters |
| Missing information | Low context recall | Chunking, embeddings, hybrid search, ingestion |
See it in code
- Needs
- Nothing: plain Python 3. The claim splitting and support labels are written by hand here; an LLM produces them in practice (e.g. with Ragas or DeepEval).
All three metrics computed for one question, showing what each catches.
question = "How many leave days do I get and can I carry them over?"
contexts = [ # what retrieval returned, in rank order
"Full-time employees get 25 days of annual leave.", # relevant
"The office closes at 6pm on Fridays.", # not relevant
"Up to 5 unused days can be carried over.", # relevant
]
relevant = [True, False, True] # labels (a judge model decides in practice)
answer_claims = [ # the answer, split into claims (by an LLM in practice)
("You get 25 days of leave.", True), # (claim, supported by the contexts?)
("You can carry over up to 5 days.", True),
("Unused days are paid out in December.", False),
]
# FAITHFULNESS: share of the answer's claims that the retrieved contexts support.
faithfulness = sum(ok for _, ok in answer_claims) / len(answer_claims)
# ANSWER RELEVANCE: does the answer address every part of the question? (judge per part)
parts_answered = {"how many days": True, "carry over": True}
answer_relevance = sum(parts_answered.values()) / len(parts_answered)
# CONTEXT PRECISION: are the relevant contexts ranked near the top? (average precision@k)
hits, precisions = 0, []
for k, rel in enumerate(relevant, start=1):
if rel:
hits += 1
precisions.append(hits / k)
context_precision = sum(precisions) / sum(relevant)
print(f"faithfulness : {faithfulness:.2f} <- one unsupported claim (a hallucination)")
print(f"answer relevance : {answer_relevance:.2f}")
print(f"context precision: {context_precision:.2f} <- an irrelevant chunk sits at rank 2")faithfulness : 0.67 <- one unsupported claim (a hallucination)
answer relevance : 1.00
context precision: 0.83 <- an irrelevant chunk sits at rank 2The answer addresses both parts of the question but adds an invented payout rule. Retrieval was decent but not perfect. Libraries such as Ragas (faithfulness, answer_relevancy, context_precision, context_recall) and DeepEval compute these with an LLM judge over a whole dataset.
A real example
Example
Diagnosing an HR assistant with four numbers
An HR assistant's answers were rated "unhelpful" by employees. The team ran RAG evals on 200 golden questions: faithfulness 0.95, answer relevance 0.90, context precision 0.58, context recall 0.71. The model was behaving well; retrieval was the problem: lots of irrelevant chunks, and some needed information missing.
They added hybrid search and a reranker, and fixed chunking of policy tables. Context precision rose to 0.84 and recall to 0.89, and "unhelpful" ratings halved. Without separate metrics they would probably have swapped the model, which wouldn't have helped.
The easy way
Start here
Start with faithfulness: for 30 answers, ask a judge model "list each claim in the answer and say whether the sources support it". The share of unsupported claims is your hallucination rate. Add context precision and recall once you have labelled relevant chunks.
When you grow: the production setup
Teams run these metrics in CI on golden sets and on sampled production traffic, calibrate the judges against human labels, track metrics per query type, combine them with retrieval metrics (Recall@k, MRR), and set alert thresholds (e.g. faithfulness below 0.9 on production samples).
Trade-offs
What you gain
- Separates retrieval problems from generation problems.
- Faithfulness gives a direct hallucination rate.
- Works with LLM judges, so it scales to large datasets.
What it costs you
- Judge-based metrics cost tokens and need calibration.
- Claim splitting and support judgements can themselves be wrong.
- Context recall needs reference answers or relevance labels.
When not to use it
- Non-RAG features (creative writing, chit-chat) where there are no sources to be faithful to.
Common mistakes
- One overall score instead of separate retrieval and generation metrics.
- Trusting library scores without checking them against human judgement.
- Measuring faithfulness without checking relevance (a grounded non-answer scores well).
- Fixing the prompt when the numbers point to retrieval.
- Evaluating only on easy questions.
Interview questions
Try answering out loud first, then open the model answer.
How do you evaluate a RAG system?
Model answer
Separately evaluate retrieval and generation. Retrieval: Recall@k and MRR on labelled relevant documents, plus context precision and recall. Generation: faithfulness (claims supported by retrieved context), answer relevance (addresses the question completely), citation validity, and correctness against reference answers where available. Use LLM judges calibrated against human labels, run on golden sets in CI and on sampled production traffic, and slice by query type.
Faithfulness vs correctness: what's the difference?
Model answer
Faithfulness checks whether the answer's claims are supported by the retrieved context, regardless of real-world truth; correctness checks whether the answer matches the true answer. A faithful answer can be incorrect if the context is wrong or outdated; an answer can be correct but unfaithful if the model used its own knowledge. Both matter: faithfulness measures grounding, correctness measures end-to-end quality.
Your RAG metrics show high faithfulness but low context recall. What do you do?
Model answer
The model is grounded but missing information, so improve retrieval: check whether the needed content is in the index and parsed correctly, adjust chunking, try hybrid search, query rewriting or multi-query retrieval, increase candidate k with reranking, and add metadata filters. Then re-measure context recall and end-to-end answer quality.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Faithfulness
- How well every claim in an answer is supported by the provided sources. Unsupported claims are hallucinations.
- Answer relevance
- Whether an answer actually addresses the question asked, every part of it, without drifting or padding.
- Context precision
- How much of the retrieved context is relevant, and whether the relevant chunks are ranked near the top.
- Context recall
- Whether the retrieved context contains everything needed to answer the question.
- Hallucination
- A confident model output that is false, made up or not supported by the given sources.
Before and after
Progress is saved in this browser only.