Area 04 // EvalsLesson 67 / 14013 min read
Error analysis
Error analysis means actually reading a sample of your system's failures, writing down what went wrong in each, grouping those notes into a few failure categories, and counting them, so you fix the biggest real problem first instead of guessing.
Think of it like this
Analogy
A GP doesn't treat "the hospital's average patient feeling worse" by guessing. They see patients, write notes on each ("chest infection", "sprained ankle", "chest infection"), and notice a pattern: a chest infection outbreak. Then they act on that.
Error analysis is seeing the patients: reading failures one by one, naming each problem, and counting which problem is most common.
The problem it solves
Teams often jump from "the quality score dropped" or "users complain" straight to a fix: a longer prompt, a bigger model, a new framework. Without looking at actual failures, they fix the wrong thing. Generic metrics (like "helpfulness") also hide what's really going wrong. Before building elaborate evals, you need to know which failures actually happen.
How it works, step by step
- Gather traces: a sample of real interactions with full detail (input, retrieved documents, tool calls, intermediate steps, output), ideally 50–100, focusing on ones marked bad by users, judges or your own review (tracing).
- Read each one and judge it: good or bad? Use one person with domain knowledge (or a small, aligned group) to keep judgements consistent.
- Open coding: for each bad one, write a short, specific note: "cited the 2023 policy", "retrieved the expenses doc for a leave question", "invented a number".
- Axial coding: group the notes into a few failure modes ("stale document", "wrong retrieval", "missed detail", "hallucination").
- Count each mode (and weigh by harm): the biggest bucket is usually your first fix.
- Find the component: is it parsing, retrieval, the prompt, the model, a tool?
- Build a targeted eval for each important mode (cases plus a check), fix, and re-measure.
- Repeat every few weeks or after big changes: usage shifts and new failure modes appear.
See it in code
- Needs
- Nothing: plain Python 3. The notes are written by a reviewer; grouping uses simple keyword rules here (people or an LLM do it in practice).
From reviewer notes to a ranked list of failure modes.
from collections import Counter
# Step 1 (open coding): read failing traces and write a short note on what went wrong.
reviewed = [
{"trace": "t01", "ok": False, "note": "cited the 2023 policy, not the current one"},
{"trace": "t02", "ok": True, "note": ""},
{"trace": "t03", "ok": False, "note": "answer missed the 5-day carry-over limit"},
{"trace": "t04", "ok": False, "note": "retrieved the expenses doc for a leave question"},
{"trace": "t05", "ok": False, "note": "used an old policy version"},
{"trace": "t06", "ok": False, "note": "wrong doc retrieved (remote work vs leave)"},
{"trace": "t07", "ok": True, "note": ""},
{"trace": "t08", "ok": False, "note": "invented a number not in the sources"},
]
# Step 2 (axial coding): group the notes into a few failure categories.
CATEGORIES = {
"stale document": ["old policy", "2023"],
"wrong retrieval": ["retrieved the", "wrong doc"],
"missed detail": ["missed"],
"hallucination": ["invented"],
}
def categorise(note: str) -> str:
return next((cat for cat, keys in CATEGORIES.items() if any(k in note for k in keys)), "other")
failures = [categorise(r["note"]) for r in reviewed if not r["ok"]]
print(f"failure rate: {len(failures)}/{len(reviewed)}")
for cat, n in Counter(failures).most_common(): # Step 3: count, then fix the biggest bucket first
print(f" {cat:16} {n} {'#' * n}")failure rate: 6/8
stale document 2 ##
wrong retrieval 2 ##
missed detail 1 #
hallucination 1 #Two of the top problems (stale documents, wrong retrieval) live in ingestion and retrieval, not in the prompt, which is where many teams would have started fixing.
A real example
Example
The chatbot that needed fresher data, not a better prompt
An insurance chatbot's satisfaction score dropped. The team's first idea was to switch to a stronger model. Instead, a product manager and an engineer spent an afternoon reading 80 thumbs-down conversations and coding them. Results: 31 were stale or superseded documents, 18 wrong retrieval, 12 genuinely hard questions, 7 tone, the rest miscellaneous.
They fixed ingestion (deleting superseded documents, adding updated-date filters), added an eval for "uses the current policy", and saw satisfaction recover. The model swap would have cost more and fixed perhaps a tenth of the problems.
The easy way
Start here
Once a week, open 30 conversations users marked as bad (or that a judge flagged). For each, write one sentence about what went wrong. Then sort the sentences into piles and count. Fix the biggest pile.
When you grow: the production setup
Trade-offs
What you gain
- Fixes target real, frequent problems instead of guesses.
- Points to the exact component (parsing, retrieval, prompt, model, tools).
- Tells you which evals are worth building.
What it costs you
- Takes focused human time, ideally from someone with domain knowledge.
- Small samples can miss rare but severe failures.
- Categories are judgement calls and need refining over time.
When not to use it
- Never skip it; scale it down (20 traces) rather than out.
Common mistakes
- Looking only at aggregate scores and never reading traces.
- Reading outputs without the retrieved context and tool calls, so the cause stays hidden.
- Starting with generic metrics before knowing the real failure modes.
- Too many categories (each with one example) to act on.
- Doing it once and never again.
Interview questions
Try answering out loud first, then open the model answer.
How do you do error analysis on an LLM application?
Model answer
Sample failing or low-scoring traces with full context (inputs, retrieval results, tool calls, outputs); have a domain expert judge each and write a specific note (open coding); cluster notes into a small set of failure modes (axial coding); count and weigh them by impact; trace each mode to the responsible component; fix the top ones; and build targeted evals to confirm fixes and prevent regressions. Repeat periodically.
Why do error analysis before building generic eval metrics?
Model answer
Generic metrics (helpfulness, coherence) often don't correspond to the failures that matter in a specific product and can stay high while users suffer. Error analysis reveals the actual failure modes, which then define targeted, meaningful evals and point to concrete fixes.
Users complain about wrong answers. How do you find the root cause?
Model answer
Collect the complained-about conversations, open their traces, and check each step: was the question understood, was the right content in the index and retrieved, was the prompt followed, did tools return correct data, was the answer faithful to sources? Code and count the failures to find the dominant cause, then fix that component and add evals.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Error analysis
- Reading real failures, noting what went wrong in each, grouping them into failure modes and counting them to decide what to fix first.
- Failure mode
- A recurring kind of mistake a system makes, such as wrong retrieval, stale documents or invented numbers.
- Open coding
- In error analysis, writing a short free-text note describing what went wrong in each failure.
- Axial coding
- In error analysis, grouping the individual failure notes into a small number of categories.
- Trace
- The full record of one request or task, made of nested timed spans for every step it took.
Before and after
Progress is saved in this browser only.