Area 07 // DeploymentLesson 113 / 14013 min read

Eval gates in CI

An eval gate is a pipeline step that runs your evaluation set against the new version and blocks the merge or release if quality metrics fall below fixed thresholds or drop compared with the current main branch, so prompt, model and retrieval changes can't silently make answers worse.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A bakery chain changes a recipe. Before it reaches every shop, a tasting panel scores the new loaf against the current one on taste, texture and rise. If any score falls below the standard, or is clearly worse than today's loaf, the recipe doesn't ship, however good it looked on paper.

The problem it solves

In an AI app, the riskiest changes often aren't code: a prompt edit, a new model version, a different chunk size, a reranker tweak. Unit tests pass, the app runs, and answers quietly get worse. Without an automatic quality check in the pipeline, you find out from users weeks later. Evals in CI introduced the idea; here you make it a hard gate on releases.

How it works, step by step

Have a golden set of representative cases with expected results (golden datasets), plus a script that computes your metrics (RAG evals, agent evals).
Define the rules:
- Absolute floors per metric (e.g. recall@3 ≥ 0.90, faithfulness ≥ 0.85, citation validity = 1.0).
- No regression versus the main branch beyond a tolerance set from measured noise.
- Optional budgets: cost per request, p95 latency, safety tests that must all pass.
Store the baseline: scores from the latest main-branch run (an artifact or a small file).
Run on the right triggers: changes to prompts, model config, retrieval, chunking, tools, guardrails and the eval set itself.
Exit non-zero on failure so the CI job fails, and make that check required for merging (GitHub Actions).
Report clearly: a table in the job log or a PR comment with each metric, threshold, baseline and change.
Keep it fast: a focused subset on PRs, the full suite nightly and before releases, cached responses where nothing changed.
Handle exceptions deliberately: if a trade-off is intended (cheaper model, slightly lower score), update the threshold in a reviewed commit, never by skipping the gate.

See it in code

Needs: Nothing: plain Python 3. In the capstone app the same logic lives in evals/run_eval.py.

A gate that checks floors and regressions against main. This pull request improves recall but drops MRR too far, so it's blocked.

eval_gate.py

import json, sys
 
# An eval gate compares this commit's scores with fixed thresholds AND with the main branch (no regressions).
THRESHOLDS = {"recall@3": 0.90, "mrr": 0.75, "faithfulness": 0.85}
MAX_DROP = 0.02                                       # allowed noise vs the main branch
 
baseline = {"recall@3": 0.96, "mrr": 0.91, "faithfulness": 0.93}   # saved from the last main-branch run
candidate = {"recall@3": 0.97, "mrr": 0.86, "faithfulness": 0.92}  # this pull request
 
failures = []
for metric, value in candidate.items():
    floor_ok = value >= THRESHOLDS[metric]
    drop = baseline[metric] - value
    regress_ok = drop <= MAX_DROP
    status = "PASS" if floor_ok and regress_ok else "FAIL"
    print(f"{status}  {metric:13} {value:.2f}  (min {THRESHOLDS[metric]:.2f}, main {baseline[metric]:.2f}, change {-drop:+.2f})")
    if status == "FAIL":
        failures.append(metric)
 
print(json.dumps({"gate": "fail" if failures else "pass", "failed": failures}))
# sys.exit(1 if failures else 0)    <- in CI a non-zero exit code blocks the merge

Output

PASS  recall@3      0.97  (min 0.90, main 0.96, change +0.01)
FAIL  mrr           0.86  (min 0.75, main 0.91, change -0.05)
PASS  faithfulness  0.92  (min 0.85, main 0.93, change -0.01)
{"gate": "fail", "failed": ["mrr"]}

MRR is still far above its floor of 0.75, yet the gate fails: the right document moved down the ranking for some questions. Only the regression rule catches that. The capstone's gate, python -m evals.run_eval, runs as its own job in the capstone workflow.

A real example

Example

The cheaper model that wasn't quite good enough

A team opened a PR switching their summariser to a model a fifth of the price. Their eval gate posted a table on the PR: faithfulness dropped from 0.94 to 0.88 (floor 0.90), and two safety cases failed. Instead of arguing, they tried the cheap model with a stricter prompt and two examples; the next run scored 0.92 with all safety cases passing, and the cost saving shipped. The gate turned a judgement call into a measured decision.

The easy way

Start here

Make your eval script print a table and sys.exit(1) if any metric is below its threshold, run it as a CI job on pull requests, and mark that job as a required check. Add the regression comparison once you store main-branch scores.

When you grow: the production setup

Teams maintain tiered suites (PR, nightly, release), baselines stored per branch, tolerances from measured variance (sometimes with multiple samples and confidence intervals), cost and latency budgets, safety and red-team regression suites, PR comments with diffs and links to failing traces, and a change process for thresholds that requires review.

Trade-offs

What you gain

Quality regressions are caught before release.
Model and prompt decisions become measurable.
Reviewers see quality, cost and latency together.

What it costs you

Model calls in CI cost money and time.
Noisy metrics can block good changes or pass bad ones.
Only as good as the golden set's coverage.

When not to use it

Don't gate on metrics you haven't validated against human judgement; a misleading metric blocks the wrong things.

Common mistakes

Gates that print results but always exit 0.
Only absolute floors, so slow decay goes unnoticed.
Not triggering on prompt and config changes.
Lowering thresholds quietly to get a PR through.
A golden set that never grows with new failure cases.

Interview questions

Try answering out loud first, then open the model answer.

How do you build an eval gate for an LLM app?

Model answer

Create a representative golden set and metric script; define absolute thresholds and a regression tolerance versus the main branch from measured noise; run on PRs touching prompts, models, retrieval or tools; exit non-zero on failure and make the check required; report a metric table on the PR; use subsets on PRs and full suites nightly; and change thresholds only through reviewed commits.

How do you deal with noisy eval results in CI?

Model answer

Measure run-to-run variance and set tolerances accordingly; use deterministic settings where the API allows; run several samples and average or use confidence intervals; increase the number of cases for key metrics; cache unchanged outputs; and rely on judges validated against human labels.

What else besides quality metrics can an eval gate check?

Model answer

Safety and guardrail regression cases (must all pass), prompt-injection resistance, cost per request and token budgets, p95 latency, output format validity, citation validity, and refusal rates, each with its own threshold.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Eval gate: A pipeline step that runs evals on a change and blocks the merge or release if quality falls below thresholds or regresses versus main.
Golden dataset: A trusted, versioned set of test inputs with expected outputs or labels, used to measure an AI system repeatably.
Regression: A machine learning task that predicts a number, such as a price or a delivery time.
Threshold: A fixed limit a metric must meet, such as recall@3 of at least 0.90, used by gates and alerts.
CI/CD: Continuous integration and continuous delivery. Automatically building, testing and deploying every change through a pipeline.

Before and after

Read first