Area 04 // EvalsLesson 65 / 14014 min read
Why evals matter
Evals are repeatable tests that score your AI system on many real examples, so you know whether a change made it better or worse instead of trusting how a few demos "feel", and they run both before release (offline) and on live traffic (online).
Think of it like this
Analogy
A chef tasting one spoonful and saying "yeah, that's better" is a vibe check. A restaurant that sends every dish on a fixed tasting menu to three tasters with a scorecard, every week, is running evals. Only the second can tell you whether last week's recipe change made the menu better overall, or fixed one dish and ruined two others.
The problem it solves
AI systems are non-deterministic and sensitive: one changed sentence in a prompt, a new model version, or a tweak to chunking can improve some answers and quietly break others. Teams that test by trying a few favourite questions ship regressions they don't notice until users complain. They also can't answer basic questions: Is the new model better? Is the cheaper model good enough? Did last month's change help?
How it works: from vibes to measurement
- Collect cases: real inputs (and, where possible, expected outputs or properties) that represent what users actually do, including hard and edge cases. This becomes your golden dataset.
- Define scoring for each case: exact match, a rule ("mentions the order ID", "under 40 words"), a metric (precision/recall, retrieval metrics), an LLM judge, or a human label.
- Run the system on every case the same way, ideally several times to see variance.
- Compute scores and track them per version: overall, and sliced by category.
- Compare versions before shipping: does the change improve the target without regressing anything else?
- Look at failures, not just the number (error analysis), and turn each new production bug into a new case.
- Automate it so it runs on every change (regression evals in CI).
How it works: offline vs online evals
| Offline evals | Online evals | |
|---|---|---|
| When | Before release, on every change | Continuously, on live traffic |
| Data | Fixed, versioned test set | Real (sampled) requests |
| Scoring | Expected answers, rules, judges, human labels | Judges on samples, user feedback, business metrics, guardrail hits |
| Answers | "Is this version better than the last?" | "How good is it for real users right now?" |
| Catches | Regressions before they ship | Drift, new kinds of questions, real-world failures |
Use both: offline evals gate releases; online evals reveal what your test set is missing, and their failures feed back into it.
Eval-driven development
Eval-driven development applies test-driven development to AI behaviour:
- Write the eval first: add cases that describe the behaviour you want (or the bug you're fixing) and a way to score them.
- Run it and watch it fail on the current system.
- Change the system (prompt, model, retrieval, tools) until the new cases pass.
- Run the whole suite to make sure nothing else regressed.
- Keep the cases forever so the behaviour can't silently break again.
See it in code
- Needs
- Nothing: plain Python 3. Both 'apps' are stubs standing in for model + prompt versions.
The smallest useful eval harness: four cases, two rules, a pass rate per version.
import re
# A tiny eval harness: test cases + a scoring rule + a pass rate. Compare two prompt versions.
CASES = [
{"input": "Refund for order A1234?", "must_include": ["A1234"], "max_words": 40},
{"input": "Do you ship to Spain?", "must_include": ["Spain"], "max_words": 40},
{"input": "Cancel my subscription", "must_include": ["cancel"], "max_words": 40},
{"input": "Is the app down?", "must_include": ["status"], "max_words": 40},
]
def app_v1(text: str) -> str: # stand-in for "model + prompt version 1"
return "Thanks for reaching out! " + "We value you. " * 12 + text
def app_v2(text: str) -> str: # stand-in for "model + prompt version 2"
if "down" in text:
return "Please check our status page for live updates."
return f"Sure - about '{text}': here is what to do next."
def score(case: dict, output: str) -> bool:
words = len(output.split())
has_all = all(re.search(w, output, re.I) for w in case["must_include"])
return has_all and words <= case["max_words"]
for name, app in [("prompt v1", app_v1), ("prompt v2", app_v2)]:
results = [score(c, app(c["input"])) for c in CASES]
print(f"{name}: {sum(results)}/{len(CASES)} passed {['PASS' if r else 'FAIL' for r in results]}")prompt v1: 0/4 passed ['FAIL', 'FAIL', 'FAIL', 'FAIL']
prompt v2: 4/4 passed ['PASS', 'PASS', 'PASS', 'PASS']v1's padded answers break the length rule on every case. Real evals have hundreds of cases and richer scoring, but the shape never changes: cases, system, score, compare.
A real example
Example
The model upgrade that wasn't
A support team was excited about a new model version that "felt smarter" in demos. Before switching, they ran their 300-case eval set. Overall quality rose 3 points, but refund questions dropped 11 points: the new model was more willing to promise refunds the policy didn't allow. They adjusted the system prompt for refunds, re-ran the eval until refund cases recovered, and then switched. Without the eval, they would have shipped a model that gave away money.
The easy way
Start here
Write down 20 real questions your users ask and what a good answer must contain. Put them in a file. Every time you change the prompt, model or retrieval, run all 20 and count passes. That's a real eval, and it beats any amount of vibe checking.
When you grow: the production setup
Mature teams keep versioned golden datasets per feature, mix code-based checks, calibrated LLM judges and periodic human review, run evals in CI with thresholds, track scores per release on dashboards, sample production traffic for online judging, and close the loop by turning every escaped bug into a test case.
Trade-offs
What you gain
- Safe, confident changes to prompts, models and retrieval.
- Objective comparisons (is the cheaper model good enough?).
- Bugs stay fixed because they become test cases.
What it costs you
- Building good datasets and scoring takes real effort.
- LLM judges and human review cost money and time.
- A narrow eval set can give false confidence.
When not to use it
- Never skip evals for anything users rely on. Only throwaway prototypes can live on vibes.
Common mistakes
- Judging changes by a few hand-picked demos.
- Tracking only one overall number and missing regressions in a category.
- Never looking at the failing cases themselves.
- Letting the eval set go stale while the product changes.
- Running evals once instead of on every change.
Interview questions
Try answering out loud first, then open the model answer.
Why are evals essential for LLM applications?
Model answer
LLM behaviour is non-deterministic and sensitive to small changes in prompts, models, retrieval and data, so manual spot checks miss regressions. Evals score a representative, versioned set of cases with consistent methods, letting you compare versions objectively, gate releases, choose models on quality versus cost, and prevent fixed bugs from returning. Online evals add visibility into real-world quality and drift.
What's the difference between offline and online evaluation?
Model answer
Offline evaluation runs a candidate system against a fixed labelled dataset before release, giving reproducible comparisons and catching regressions. Online evaluation measures quality on live traffic through sampled LLM-judge scoring, user feedback, guardrail and error rates and business metrics, revealing distribution shift and failures the test set lacks. Online findings feed the offline set.
What is eval-driven development?
Model answer
Treating evals like tests in TDD: before changing an AI system, add cases that define the desired behaviour or reproduce a bug, with scoring; confirm they fail; iterate on prompts, models or retrieval until they pass; run the full suite for regressions; and keep the cases permanently. It turns vague improvement into measurable, protected progress.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Eval
- A repeatable test of an AI system's quality on a set of examples with a scoring method.
- Vibe check
- Judging an AI change by trying a few examples and going with how it feels, instead of measuring on a test set.
- Offline eval
- An evaluation run before release on a fixed, versioned test set, used to compare versions and catch regressions.
- Online eval
- Measuring quality on live production traffic, through sampled judge scores, user feedback and business metrics.
- Eval-driven development
- Writing the eval for a desired behaviour first, then changing the AI system until it passes, and keeping the cases forever.
- Golden dataset
- A trusted, versioned set of test inputs with expected outputs or labels, used to measure an AI system repeatably.
Before and after
Progress is saved in this browser only.