Skip to content
← All cheat sheets

Cheat sheet · Area 04

Evals

Why evals matter

Evals are repeatable tests that score your AI system on many real examples, so you know whether a change made it better or worse instead of trusting how a few demos "feel", and they run both before release (offline) and on live traffic (online).

  • Vibes = trying a few prompts and deciding it 'feels better'. Evals = scoring many cases the same way every time.
  • An eval = dataset + system under test + scoring method + a number you track.
  • Offline evals: before release, on a fixed test set. Online evals: on real traffic (sampled judges, user feedback, metrics).
  • Eval-driven development: write the eval first, then change prompts/models/retrieval until it passes.
  • Without evals you can't safely change prompts, models or retrieval.

Golden datasets and synthetic test data

A golden dataset is a trusted, versioned collection of real test cases with expected answers or properties that you score every version against, and synthetic test data uses an LLM to generate extra cases (variations, edge cases, rare situations) to fill the gaps.

  • Golden case = input + expected output or properties + tags (category, difficulty) + source.
  • Start from real user inputs; include easy, hard, edge and unanswerable cases.
  • Version it, review it, and grow it from production failures.
  • Synthetic data: LLM-generated variations and new cases. Great for coverage; must be reviewed; never only synthetic.
  • Keep a held-out set you never tune on, to avoid overfitting your prompts to the evals.

Error analysis

Error analysis means actually reading a sample of your system's failures, writing down what went wrong in each, grouping those notes into a few failure categories, and counting them, so you fix the biggest real problem first instead of guessing.

  • Read real traces (inputs, retrieved context, tool calls, outputs), not just scores.
  • Open coding: write a short free-text note on what went wrong in each failure.
  • Axial coding: group notes into a handful of failure modes.
  • Count them; fix the biggest bucket; build an eval for each important failure mode.
  • Repeat regularly: new failure modes appear as usage changes.

Precision, recall and F1

Precision is how many of the things you flagged were really right, recall is how many of the right things you managed to flag, and F1 balances the two; together they tell the truth about classifiers, guardrails and retrieval when accuracy alone would mislead you.

  • Confusion matrix: TP (correctly flagged), FP (wrongly flagged), FN (missed), TN (correctly left alone).
  • Precision = TP / (TP + FP): when I say yes, how often am I right?
  • Recall = TP / (TP + FN): of all the real yeses, how many did I catch?
  • F1 = 2·P·R / (P + R): high only when both are high.
  • Accuracy lies when one class is rare: 'never flag' can be 95% accurate and catch nothing.

LLM as a judge, pairwise comparison and rubrics

An LLM judge is a model prompted to grade another model's output, either against a rubric of clear criteria or by comparing two outputs side by side, which lets you evaluate open-ended answers at scale, as long as you check the judge agrees with humans.

  • Judge = a model + a grading prompt + structured output (verdict + reason).
  • Rubric scoring: several specific pass/fail criteria (grounded, complete, concise) beat one vague 1–10 score.
  • Pairwise: 'Which answer is better, A or B?' Run twice with the order swapped to cancel position bias.
  • Known biases: position, length (longer looks better), self-preference, leniency.
  • Calibrate: compare judge vs human labels (agreement, Cohen's kappa) before trusting it.

Human labeling and user feedback loops

Human labeling is people with clear guidelines judging AI outputs to create trusted ground truth, and user feedback loops collect signals from real users (thumbs, edits, complaints) and route them back into fixes, eval sets and training data.

  • Humans are the ground truth for evals and judges; give them written guidelines with examples.
  • Measure inter-annotator agreement (Cohen's kappa); disagreements reveal unclear guidelines.
  • User feedback: explicit (thumbs, ratings, comments) and implicit (copy, retry, edit, abandon, escalate).
  • Close the loop: feedback → triage → error analysis → fix + new golden cases → measure again.
  • Feedback is biased (angry users vote more); use it as a signal, not a score.

RAG evals: faithfulness, relevance and context precision

RAG systems are evaluated by checking the answer is faithful to the retrieved sources (no invented claims), relevant to the question (answers what was asked), and that retrieval put useful context at the top (context precision), so you can tell whether a bad answer came from retrieval or generation.

  • Faithfulness = supported claims ÷ all claims in the answer. Low = hallucination.
  • Answer relevance = does the answer address the question (all its parts), without padding?
  • Context precision = are the retrieved chunks relevant, and ranked near the top?
  • Context recall = did retrieval find everything needed to answer? (needs reference answers or labels)
  • Low faithfulness → fix generation/prompt. Low context precision/recall → fix retrieval.

Agent evals: tool calls and trajectories

Agents are evaluated not only on their final answer but on how they got there, checking whether they called the right tools with the right arguments (tool call accuracy) and whether the whole sequence of steps was sensible, safe and efficient (trajectory evals).

  • Outcome evals: was the final result right? Necessary but not enough.
  • Tool call accuracy: right tool, right arguments, at the right time (vs expected calls).
  • Trajectory evals: order, efficiency (steps, tokens), recovery from errors, no forbidden actions.
  • Score from traces; use exact checks where possible and judges for the rest.
  • Run each case several times: agents vary between runs.

Regression evals in CI and A/B testing prompts

Regression evals run your eval suite automatically on every change and block the change if quality drops, and A/B testing sends a share of real users to a new prompt or model to measure, with statistics, whether it actually performs better in production.

  • Regression eval: run the suite on every PR; compare with the main branch's baseline; fail if a metric drops beyond a tolerance.
  • Keep CI evals fast and cheap: a focused subset on every PR, the full suite nightly or before release.
  • A/B test: split real traffic between versions, measure a target metric, check significance before deciding.
  • Decide the metric, sample size and duration BEFORE starting; don't stop early when it looks good.
  • Offline evals gate merges; A/B tests confirm real-world impact.

Tracing and OpenTelemetry for LLMs

Tracing records every step of a request (retrieval, each model call with its prompt, tokens and latency, each tool call) as a tree of timed spans, and OpenTelemetry is the open standard for producing and shipping those traces, with shared GenAI attribute names so any tool can read them.

  • Trace = one request end to end. Span = one step inside it (with start/end time, parent, attributes).
  • For LLM apps record: prompts/inputs, outputs, model, tokens, cost, latency, tool calls, retrieved docs, errors, user/session, versions.
  • OpenTelemetry (OTel): vendor-neutral API + SDK + OTLP protocol for traces, metrics and logs.
  • GenAI semantic conventions: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens (status: Development, Nov 2026).
  • Redact secrets and personal data before traces leave your system.

Langfuse and observability dashboards

Langfuse is an open-source LLM engineering platform (tracing, prompt management, evaluation and dashboards) you can use in the cloud or self-host, and observability dashboards turn the traces it collects into the numbers you watch every day, such as volume, quality, latency, cost and errors.

  • Langfuse: traces + sessions + scores (evals, feedback) + prompt management + datasets + dashboards. Open source (MIT core); cloud or self-host.
  • Python SDK v5 (OpenTelemetry-based): @observe, get_client(), propagate_attributes, score_current_trace (as of Nov 2026).
  • Env vars: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL.
  • Dashboard essentials: requests, error rate, p50/p95 latency, tokens and cost (by model, user, feature), quality scores, feedback.
  • Alternatives: LangSmith, Arize Phoenix, Braintrust, Helicone, Datadog/Grafana via OpenTelemetry.

P95 latency and cost per request

P95 latency is the response time that 95% of requests beat, which shows what your slowest real users experience far better than an average, and cost per request adds up every token, model call, tool and retry behind one user request so you know what the feature really costs.

  • Averages hide the slow tail. Report p50 (typical), p95 and p99 (worst common experiences).
  • p95 = 95% of requests were faster than this. One slow step in many requests drags it up.
  • For chat, also track time to first token (TTFT): how long until text starts appearing.
  • Cost per request = sum over ALL model calls (rewrite, answer, judge, retries) of tokens × price, plus tools and infra.
  • Track both per feature, per model and per tenant; alert on changes.

Eval tools at a glance

Eval tools package the common pieces (datasets, scorers, LLM judges, experiment tracking and CI integration) so you don't build them from scratch; Ragas and DeepEval are Python libraries, promptfoo is a config-driven CLI, and LangSmith, Braintrust and Langfuse are platforms that combine evals with tracing.

  • All eval tools share one shape: dataset × system under test × scorers → report → compare versions.
  • Ragas: RAG and agent metrics (faithfulness, context precision/recall, answer relevancy) in Python.
  • DeepEval: pytest-style LLM unit tests with many built-in metrics.
  • promptfoo: YAML config + CLI to compare prompts/models with assertions; easy in CI; also red-teaming.
  • Platforms (LangSmith, Braintrust, Langfuse): datasets, experiments, judges, human review, tracing, dashboards.
  • Start with a plain script; adopt a tool when you need its metrics, UI or collaboration.