Area 04 // EvalsLesson 77 / 14013 min read

Eval tools at a glance

Eval tools package the common pieces (datasets, scorers, LLM judges, experiment tracking and CI integration) so you don't build them from scratch; Ragas and DeepEval are Python libraries, promptfoo is a config-driven CLI, and LangSmith, Braintrust and Langfuse are platforms that combine evals with tracing.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

You can bake bread with flour, water and your hands, or buy a stand mixer, a proving drawer and a bread machine. The bread is the same idea either way; the tools save time at scale, add consistency, and some are better for certain loaves. Eval tools are the kitchen gadgets of evaluation.

The problem it solves

You now know the eval ideas: golden datasets, scorers, judges, RAG and agent metrics, CI gates, tracing. Building all of it yourself (judge prompts, metric implementations, result storage, comparison views, annotation UIs) takes weeks, and teams often re-implement the same metrics slightly wrongly. Many tools exist; knowing what each is good at saves time and avoids lock-in to the wrong one.

How it works, step by step

Recognise the common shape: every tool has (a) a dataset of cases, (b) a task / system under test (your function, prompt or app), (c) scorers (code checks, metrics, LLM judges), and (d) a report or experiment you compare across versions.
Choose by need (as of November 2026):

Tool	Type	Strengths	Good when
Ragas	Python library (open source)	RAG metrics (faithfulness, answer relevancy, context precision/recall), agent and test-set generation utilities	You need standard RAG metrics in code
DeepEval	Python library (open source)	pytest-style tests, many built-in metrics (G-Eval, hallucination, RAG, agent), synthetic data	You want LLM evals to feel like unit tests
promptfoo	CLI + YAML (open source)	Compare prompts and models side by side, assertions, CI-friendly, red-teaming	You want config-driven comparisons and quick CI checks
LangSmith	Platform (LangChain)	Datasets, experiments, judges, annotation, tracing, tight LangChain/LangGraph integration	You use the LangChain ecosystem
Braintrust	Platform	Experiments and scoring, playground, logging, CI	You want an eval-first platform with a strong UI
Langfuse	Open-source platform	Tracing plus datasets, experiments, judges, annotation, prompt management	You want evals tied to self-hostable tracing (Langfuse)
OpenAI / Anthropic eval features	Provider tools	Evals and graders inside the provider console	You mostly use one provider

Start with a plain script (tiny eval) while your dataset and scoring settle down.
Adopt a library when you need well-tested metrics (Ragas, DeepEval) or easy comparisons (promptfoo).
Adopt a platform when several people need to review results, label data, track experiments over time and link evals to production traces.
Keep your data portable: datasets as JSONL in Git, scorers as code, so switching tools is possible.
Validate tool metrics against human judgement before trusting them (LLM as a judge).

Different tools, same pipeline.

See it in code

Needs: Nothing for the plain-Python version. The promptfoo config needs Node.js (npx promptfoo) and a provider key.

The shape every tool implements, in plain Python: dataset, system, scorers, report.

eval_shape.py

import json
 
# Most eval tools share one shape: dataset x system under test x metrics -> a report.
DATASET = [
    {"input": "Leave days?", "expected": "25"},
    {"input": "Carry-over limit?", "expected": "5"},
    {"input": "CEO salary?", "expected": "don't know"},
]
 
def system_under_test(q: str) -> str:              # your app (stubbed here)
    return {"Leave days?": "You get 25 days.", "Carry-over limit?": "Up to 5 days.",
            "CEO salary?": "It's 2 million."}[q]
 
METRICS = {                                         # simple code metrics; tools add LLM-judged ones
    "contains_expected": lambda case, out: case["expected"].lower() in out.lower(),
    "short": lambda case, out: len(out.split()) <= 20,
}
 
report = []
for case in DATASET:
    out = system_under_test(case["input"])
    report.append({"input": case["input"], **{m: f(case, out) for m, f in METRICS.items()}})
 
for row in report:
    print(json.dumps(row))
for m in METRICS:
    print(f"{m}: {sum(r[m] for r in report)}/{len(report)}")

Output

{"input": "Leave days?", "contains_expected": true, "short": true}
{"input": "Carry-over limit?", "contains_expected": true, "short": true}
{"input": "CEO salary?", "contains_expected": false, "short": true}
contains_expected: 2/3
short: 3/3

The same idea as a promptfoo config, comparing two models with assertions (run with npx promptfoo eval):

promptfooconfig.yaml

prompts:
  - "Answer from the HR policy only. Question: {{question}}"
providers:
  - anthropic:messages:claude-haiku-5-5
  - openai:responses:gpt-6-luna
tests:
  - vars: { question: "How many leave days do I get?" }
    assert:
      - type: icontains
        value: "25"
  - vars: { question: "What is the CEO's salary?" }
    assert:
      - type: llm-rubric
        value: "Politely says it doesn't know or can't share that."

A real example

Example

Three tools at three stages

A startup's eval tooling grew with the team. Month one: a 60-line Python script over a JSONL golden set, run by hand. Month three: Ragas metrics for faithfulness and context precision added to the script, which now ran in CI as an eval gate. Month six: with four engineers and two domain reviewers, they moved datasets and experiments into a platform for shared history, annotation queues and links from failing cases to production traces. The JSONL datasets and scorer code moved with them unchanged.

The easy way

Start here

Keep using a plain script until it hurts. When you need RAG metrics, add Ragas or DeepEval to the script. When you want quick model or prompt comparisons in CI, try promptfoo. When several people need to review results together, pick a platform.

When you grow: the production setup

Teams standardise on one platform for experiments and review plus libraries for specific metrics, keep datasets and scorers in version control, run evals in CI and on production samples, calibrate any LLM-judged metric against humans, and avoid lock-in by keeping the eval pipeline's inputs and outputs in open formats.

Trade-offs

What you gain

Ready-made metrics, judges and reports save weeks.
Platforms add history, collaboration, annotation and trace links.
CLI and pytest-style tools fit naturally into CI.

What it costs you

Metrics can be black boxes; they still need validation.
Platforms add cost and data-sharing questions.
Tool churn: APIs and names change quickly.

When not to use it

Very early prototypes: a script and 20 cases are enough.
When a tool's built-in metric doesn't match what quality means for your product: write your own scorer.

Common mistakes

Adopting a heavy platform before having a real dataset.
Trusting built-in metric scores without checking them against human judgement.
Locking datasets inside one tool with no export.
Running dozens of generic metrics instead of the few that match your failure modes.
Not wiring the tool into CI, so it only runs when someone remembers.

Interview questions

Try answering out loud first, then open the model answer.

Compare some LLM evaluation tools.

Model answer

Ragas focuses on RAG and agent metrics computed with LLM judges; DeepEval offers pytest-style LLM tests with many built-in metrics; promptfoo is a YAML-driven CLI for comparing prompts and models with assertions and red-teaming, easy in CI; LangSmith, Braintrust and Langfuse are platforms combining datasets, experiments, judges, human annotation, tracing and dashboards. Choose by whether you need metrics, comparisons, collaboration or trace integration, and keep data portable.

How do you decide when to adopt an eval platform?

Model answer

When the team outgrows scripts: several people need to review and label results, you need experiment history and comparisons across many runs, you want evals linked to production traces and prompt versions, or you need annotation queues and dashboards. Before that, a version-controlled dataset and scorer code with a CI gate is enough.

What risks come with using a tool's built-in eval metrics?

Model answer

The metric's definition may not match your product's notion of quality, LLM-judged metrics have biases and vary by judge model and prompt, scores may change when the tool updates, and opaque implementations make failures hard to interpret. Validate against human labels on your data, pin versions, and complement with custom scorers derived from error analysis.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Eval: A repeatable test of an AI system's quality on a set of examples with a scoring method.
LLM as a judge: Using a model, with a grading prompt and criteria, to score another model's outputs at scale.
Golden dataset: A trusted, versioned set of test inputs with expected outputs or labels, used to measure an AI system repeatably.
Regression eval: Running an eval suite on every change and failing it if quality drops below the previous baseline.

Before and after

Read first