Area 04 // EvalsLesson 68 / 14013 min read
Precision, recall and F1
Precision is how many of the things you flagged were really right, recall is how many of the right things you managed to flag, and F1 balances the two; together they tell the truth about classifiers, guardrails and retrieval when accuracy alone would mislead you.
Think of it like this
Analogy
A fishing boat wants to catch salmon and nothing else.
- Precision: of everything in the net, how much is salmon (not boots and dolphins)?
- Recall: of all the salmon in the lake, how many ended up in the net?
A tiny net in one perfect spot has great precision but poor recall. A giant net catches every salmon but also lots of junk: great recall, poor precision. F1 rewards a net that does well at both.
The problem it solves
Many AI components make yes/no decisions: is this message unsafe (guardrails), is this ticket urgent (classification), is this chunk relevant (retrieval and context precision), does this answer contain a hallucination (judges)? "Accuracy" is the obvious score, and it's often badly misleading, because the interesting class is usually rare. You need measures that separately count the two kinds of mistakes, because they have very different costs.
How it works, step by step
- Build the confusion matrix on a labelled test set:
- True positive (TP): flagged, and really positive.
- False positive (FP): flagged, but actually negative (a false alarm).
- False negative (FN): not flagged, but actually positive (a miss).
- True negative (TN): not flagged, and really negative.
- Precision = TP / (TP + FP): of the things you flagged, the share that were right. Low precision = annoying false alarms.
- Recall = TP / (TP + FN): of the real positives, the share you caught. Low recall = dangerous misses.
- F1 = 2 × P × R / (P + R): one number that's high only when both are high.
- Accuracy = (TP + TN) / everything: fine when classes are balanced, misleading when they aren't.
- Choose what matters: safety filters and fraud detection usually prioritise recall (don't miss threats); spam folders or auto-actions often prioritise precision (don't annoy or harm users with false alarms).
- Tune the threshold: most classifiers output a score; a higher threshold raises precision and lowers recall, and vice versa. Pick the point that fits the costs.
- Same ideas elsewhere: Recall@k for retrieval, context precision for RAG, judge agreement for LLM-as-a-judge.
| Predicted yes | Predicted no | |
|---|---|---|
| Actually yes | TP | FN (missed) |
| Actually no | FP (false alarm) | TN |
See it in code
- Install
pip install scikit-learn- Needs
- Python 3.10+
A risky-message detector scored four ways, and the "never flag" model that shows why accuracy misleads.
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
# A classifier that flags risky messages. 1 = risky, 0 = fine. 20 test messages.
truth = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
predicted = [1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
tn, fp, fn, tp = confusion_matrix(truth, predicted).ravel()
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
print(f"accuracy : {(tp + tn) / len(truth):.2f}") # looks great...
print(f"precision: {precision_score(truth, predicted):.2f} (of flagged, how many were really risky)")
print(f"recall : {recall_score(truth, predicted):.2f} (of risky, how many we caught)")
print(f"F1 : {f1_score(truth, predicted):.2f} (balance of the two)")
always_fine = [0] * len(truth) # a useless model that never flags
print(f"'never flag' accuracy: {sum(t == p for t, p in zip(truth, always_fine)) / len(truth):.2f}, "
f"recall: {recall_score(truth, always_fine):.2f}")TP=3 FP=2 FN=2 TN=13
accuracy : 0.80
precision: 0.60 (of flagged, how many were really risky)
recall : 0.60 (of risky, how many we caught)
F1 : 0.60 (balance of the two)
'never flag' accuracy: 0.75, recall: 0.0080% accuracy sounds good, yet the detector misses 2 of 5 risky messages, and a model that never flags anything still scores 75%. Precision and recall show the real picture.
A real example
Example
Tuning a self-harm detector for a mental-health app
A wellbeing app runs a classifier on messages to detect possible self-harm risk and show crisis resources. With the default threshold, precision was 0.85 and recall 0.70: three in ten at-risk messages were missed. The team lowered the threshold until recall reached 0.95, accepting precision of 0.55 (more false alarms), because showing resources unnecessarily was a small cost and missing someone at risk was not. They monitor both numbers weekly on a reviewed sample.
The easy way
Start here
For any yes/no component, label 100 real examples, then write the four counts (TP, FP, FN, TN) on paper. Compute precision and recall. Ask: which mistake hurts more here? That tells you which number to push up.
When you grow: the production setup
Teams report precision, recall and F1 per class and per slice, plot precision-recall curves to choose thresholds, weigh errors by business cost, monitor the metrics on production samples (they drift), and use macro or weighted averages for multi-class problems.
Trade-offs
What you gain
- Separates the two kinds of mistakes, which usually have very different costs.
- Works for classifiers, guardrails, retrieval and LLM judges.
- Threshold tuning gives direct control of the trade-off.
What it costs you
- Needs labelled examples, including enough rare positives.
- Single numbers hide behaviour on sub-groups.
- Pushing one up usually pushes the other down.
Common mistakes
- Reporting accuracy on imbalanced data.
- Reporting precision without recall (or the reverse).
- Ignoring which error is more costly for the product.
- Too few positive examples in the test set to measure recall reliably.
- Never revisiting the threshold as data changes.
Interview questions
Try answering out loud first, then open the model answer.
Explain precision, recall and F1 with an example.
Model answer
For a fraud detector: precision is the share of flagged transactions that are actually fraud (low precision means many false alarms annoying customers); recall is the share of all fraud that gets flagged (low recall means missed fraud); F1 is their harmonic mean, high only when both are high. Accuracy is misleading because fraud is rare: never flagging gives near-perfect accuracy and zero recall.
How do you choose between optimising precision and recall?
Model answer
By the relative cost of false positives and false negatives. Safety, medical and fraud screening usually favour recall, with a second step or human review to handle false positives; automatic actions that affect users (auto-blocking, auto-refunds) favour precision. Then tune the decision threshold on a precision-recall curve to the chosen operating point.
How do precision and recall apply to RAG evaluation?
Model answer
Retrieval recall (Recall@k) measures whether the relevant chunks were retrieved at all; context precision measures how much of what was retrieved (and how highly ranked) is relevant. Low recall causes missing information; low precision wastes context and distracts the model. Faithfulness can also be framed as the precision of the answer's claims with respect to the sources.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Precision
- Of the items a system flagged or returned, the share that were actually correct or relevant. TP / (TP + FP).
- Recall
- The share of the truly relevant items that a search (or classifier) actually finds.
- F1 score
- The harmonic mean of precision and recall; high only when both are high.
- Confusion matrix
- A table counting true positives, false positives, false negatives and true negatives for a classifier.
- False positive
- Flagging something as positive when it isn't (a false alarm).
- False negative
- Missing something that really was positive.
- Accuracy
- The share of predictions that are correct. Misleading when one class is rare; use precision and recall then.
Before and after
Progress is saved in this browser only.