Area 04 // EvalsLesson 73 / 14015 min read
Regression evals in CI and A/B testing prompts
Regression evals run your eval suite automatically on every change and block the change if quality drops, and A/B testing sends a share of real users to a new prompt or model to measure, with statistics, whether it actually performs better in production.
Think of it like this
Analogy
A factory has two checks. On the production line, every batch passes an automatic inspection station: if a measurement falls outside tolerance, the batch is stopped (regression evals in CI). And when they consider a new recipe, they sell it in half the shops for a month and compare sales with the old recipe in the other half (an A/B test).
The problem it solves
Once you have evals, people still have to remember to run them, and they don't, especially for "tiny" prompt edits. Regressions slip into production. And even with great offline evals, you can't be sure a change helps real users until you measure it with real users; offline evals are a prediction, not proof.
How it works: regression evals in CI
- Choose the CI suite: a fast, focused subset of the golden set (say 50–200 cases) covering critical behaviours, plus deterministic checks (format, citations, retrieval metrics). Run the full suite nightly or before releases.
- Store a baseline: the scores of the current main branch, per metric (and per slice).
- On every pull request, run the suite against the changed system (GitHub Actions).
- Compare with the baseline: fail if any important metric drops more than a tolerance set from measured run-to-run noise; optionally require improvement on the targeted metric.
- Report in the PR: a table of metrics, the cases that changed from pass to fail, links to traces.
- Control cost and time: cache unchanged outputs, run judges on a cheaper model where calibrated, parallelise, and keep the PR suite under a few minutes.
- Update the baseline when changes merge, and version both the dataset and the baseline. This is the eval gate the Deployment area builds.
How it works: A/B testing prompts
- Decide the question and metric before starting: thumbs-up rate, task completion, escalation rate, conversion, plus guardrail metrics that must not get worse (cost, latency, complaints).
- Compute the sample size needed to detect the effect you care about.
- Split traffic randomly and consistently (the same user always sees the same version), often 50/50 or a small share for risky changes, using feature flags.
- Log the version with every request and outcome (tracing).
- Run until the planned sample size; don't stop the moment one version looks ahead.
- Test for significance (e.g. a two-proportion z-test for rates) and check guardrail metrics.
- Decide: roll out, roll back, or iterate. Feed surprising failures back into the golden set.
See it in code
- Needs
- Nothing: plain Python 3.
A regression gate against a baseline, and a significance test for an A/B result.
import math
# REGRESSION EVAL GATE: compare the candidate's scores with the main branch's baseline.
baseline = {"faithfulness": 0.93, "answer_relevance": 0.88, "recall@5": 0.86}
candidate = {"faithfulness": 0.94, "answer_relevance": 0.81, "recall@5": 0.87}
TOLERANCE = 0.02 # allow small noise, block real drops
failed = [m for m in baseline if candidate[m] < baseline[m] - TOLERANCE]
for m in baseline:
print(f"{m:17} {baseline[m]:.2f} -> {candidate[m]:.2f} {'REGRESSION' if m in failed else 'ok'}")
# A/B TEST: did prompt B really get more thumbs-up than prompt A in production?
def two_proportion_z(success_a, n_a, success_b, n_b):
p_a, p_b = success_a / n_a, success_b / n_b
pooled = (success_a + success_b) / (n_a + n_b)
se = math.sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se
p_value = math.erfc(abs(z) / math.sqrt(2)) # two-sided
return p_a, p_b, p_value
p_a, p_b, p = two_proportion_z(620, 1000, 668, 1000)
print(f"A/B: A {p_a:.1%} vs B {p_b:.1%}, p-value {p:.3f} -> {'B wins' if p < 0.05 else 'not significant yet'}")
print("gate:", "FAIL - block the merge" if failed else "PASS")
# In CI, end with: sys.exit(1 if failed else 0) -> a non-zero exit code fails the jobfaithfulness 0.93 -> 0.94 ok
answer_relevance 0.88 -> 0.81 REGRESSION
recall@5 0.86 -> 0.87 ok
A/B: A 62.0% vs B 66.8%, p-value 0.025 -> B wins
gate: FAIL - block the mergeThe candidate improved two metrics a little but dropped answer relevance by 7 points, so the gate blocks it. The RAG capstone's evals/run_eval.py is exactly this pattern, wired into CI in the eval gates lesson.
A real example
Example
The prompt tweak that CI caught
An engineer shortened a support bot's system prompt to cut tokens. Locally, five favourite questions looked fine. The PR's eval job ran 150 golden cases: token use dropped 18%, but "asks for order number when missing" fell from 96% to 71%, because a removed sentence covered it. The PR was blocked with the failing cases listed. The engineer restored that sentence, kept the rest of the savings, and merged.
A month later, a new greeting style passed CI and went to an A/B test at 50/50 for two weeks. Satisfaction was flat (p = 0.4) but escalations rose slightly, so they kept the old style.
The easy way
Start here
Make your eval script exit with code 1 when a key metric is below a threshold, and run it in CI on every pull request (GitHub Actions). For A/B tests, start by logging the prompt version on every request and comparing thumbs-up rates per version over a fixed period.
When you grow: the production setup
Teams run tiered eval suites (fast per-PR, full nightly, release suites), cache outputs, post rich PR comments, version baselines and datasets, use experimentation platforms with proper randomisation and sample-size planning, guard metrics, sequential testing methods, and automatic rollback on guardrail breaches (canary releases).
Trade-offs
What you gain
- Quality regressions are blocked automatically, like failing tests.
- A/B tests measure real user impact, not just predictions.
- Decisions become evidence-based and repeatable.
What it costs you
- Eval runs in CI cost tokens and time.
- Noise can cause flaky gates if tolerances aren't set from real variance.
- A/B tests need enough traffic and time; small products may never reach significance.
When not to use it
- Tiny traffic: A/B tests won't be conclusive; rely on offline evals and careful staged rollouts.
Common mistakes
- Evals that exist but don't run automatically.
- Gates with zero tolerance that fail on random noise.
- Only an overall score, so slice regressions slip through.
- Stopping A/B tests early when the result looks good.
- Ignoring cost and latency in A/B decisions.
Interview questions
Try answering out loud first, then open the model answer.
How do you prevent prompt changes from degrading quality?
Model answer
Treat prompts as code: version them, require review, and run a regression eval suite in CI on every change against a stored baseline, failing the build if key metrics drop beyond a noise-based tolerance, with a report of newly failing cases. Run broader suites nightly and before release, then roll out with canaries or A/B tests and monitor online metrics with rollback.
How would you design an A/B test for a new prompt?
Model answer
Define the primary metric and guardrail metrics, the minimum effect worth detecting, and the required sample size; randomise users consistently with feature flags; log versions and outcomes per request; run to the planned size without peeking-based early stops; test significance and check guardrails; then roll out, iterate or revert, and add discovered failures to the golden set.
Offline evals say B is better but the A/B test shows no difference. Why?
Model answer
The golden set may not represent real traffic, the metric offline may not drive the user metric, the improvement may be too small to detect with the traffic available, novelty or seasonal effects may interfere, or the online metric may be noisy. Investigate with error analysis on production traces and update the eval set to better match reality.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Regression eval
- Running an eval suite on every change and failing it if quality drops below the previous baseline.
- CI/CD
- Continuous integration and continuous delivery. Automatically building, testing and deploying every change through a pipeline.
- A/B test
- Randomly splitting real users between two versions and comparing a metric to see which performs better.
- Statistical significance
- Evidence that a measured difference is unlikely to be due to random chance, often judged by a p-value below 0.05.
- Eval
- A repeatable test of an AI system's quality on a set of examples with a scoring method.
Before and after
Progress is saved in this browser only.