Area 04 of 08

Evals

How to know if your AI app is actually good: test sets, judges, RAG and agent metrics, evals in CI, tracing and the numbers to watch in production.

Printable cheat sheet for this area

1. 065

   Why evals matter

   Evals are repeatable tests that score your AI system on many real examples, so you know whether a change made it better or worse instead of trusting how a few demos "feel", and they run both before release (offline) and on live traffic (online).

   CoversWhy evals matter (vibes vs measurement) · Offline vs online evals · Eval-driven development

2. 066

   Golden datasets and synthetic test data

   A golden dataset is a trusted, versioned collection of real test cases with expected answers or properties that you score every version against, and synthetic test data uses an LLM to generate extra cases (variations, edge cases, rare situations) to fill the gaps.

   CoversGolden Datasets · Synthetic Test Data

3. 067

   Error analysis

   Error analysis means actually reading a sample of your system's failures, writing down what went wrong in each, grouping those notes into a few failure categories, and counting them, so you fix the biggest real problem first instead of guessing.

4. 068

   Precision, recall and F1

   Precision is how many of the things you flagged were really right, recall is how many of the right things you managed to flag, and F1 balances the two; together they tell the truth about classifiers, guardrails and retrieval when accuracy alone would mislead you.

5. 069

   LLM as a judge, pairwise comparison and rubrics

   An LLM judge is a model prompted to grade another model's output, either against a rubric of clear criteria or by comparing two outputs side by side, which lets you evaluate open-ended answers at scale, as long as you check the judge agrees with humans.

   CoversLLM as a Judge · Pairwise Comparison · Rubric Scoring

6. 070

   Human labeling and user feedback loops

   Human labeling is people with clear guidelines judging AI outputs to create trusted ground truth, and user feedback loops collect signals from real users (thumbs, edits, complaints) and route them back into fixes, eval sets and training data.

   CoversHuman Labeling · User Feedback Loops

7. 071

   RAG evals: faithfulness, relevance and context precision

   RAG systems are evaluated by checking the answer is faithful to the retrieved sources (no invented claims), relevant to the question (answers what was asked), and that retrieval put useful context at the top (context precision), so you can tell whether a bad answer came from retrieval or generation.

   CoversFaithfulness · Answer Relevance · Context Precision

8. 072

   Agent evals: tool calls and trajectories

   Agents are evaluated not only on their final answer but on how they got there, checking whether they called the right tools with the right arguments (tool call accuracy) and whether the whole sequence of steps was sensible, safe and efficient (trajectory evals).

   CoversTool Call Accuracy · Trajectory Evals

9. 073

   Regression evals in CI and A/B testing prompts

   Regression evals run your eval suite automatically on every change and block the change if quality drops, and A/B testing sends a share of real users to a new prompt or model to measure, with statistics, whether it actually performs better in production.

   CoversRegression Evals in CI · A/B Testing Prompts

10. 074

    Tracing and OpenTelemetry for LLMs

    Tracing records every step of a request (retrieval, each model call with its prompt, tokens and latency, each tool call) as a tree of timed spans, and OpenTelemetry is the open standard for producing and shipping those traces, with shared GenAI attribute names so any tool can read them.

    CoversTracing · OpenTelemetry for LLMs

11. 075

    Langfuse and observability dashboards

    Langfuse is an open-source LLM engineering platform (tracing, prompt management, evaluation and dashboards) you can use in the cloud or self-host, and observability dashboards turn the traces it collects into the numbers you watch every day, such as volume, quality, latency, cost and errors.

    CoversLangfuse · Observability dashboards

12. 076

    P95 latency and cost per request

    P95 latency is the response time that 95% of requests beat, which shows what your slowest real users experience far better than an average, and cost per request adds up every token, model call, tool and retry behind one user request so you know what the feature really costs.

    CoversP95 Latency · Cost per Request

13. 077

    Eval tools at a glance

    Eval tools package the common pieces (datasets, scorers, LLM judges, experiment tracking and CI integration) so you don't build them from scratch; Ragas and DeepEval are Python libraries, promptfoo is a config-driven CLI, and LangSmith, Braintrust and Langfuse are platforms that combine evals with tracing.