Area 04 // EvalsLesson 66 / 14015 min read

Golden datasets and synthetic test data

A golden dataset is a trusted, versioned collection of real test cases with expected answers or properties that you score every version against, and synthetic test data uses an LLM to generate extra cases (variations, edge cases, rare situations) to fill the gaps.

On this page

Think of it like this
The problem it solves
How it works: golden datasets
How it works: synthetic test data
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A driving examiner uses a standard test route: the same tricky roundabout, the same hill start, the same school zone, for every candidate. That's how they compare drivers fairly. Over the years they add new hazards to the route when accidents show something was missing.

A golden dataset is your AI system's standard test route. Synthetic data is the examiner sketching extra practice scenarios ("what if a ball rolls into the road?") to make sure rare hazards are covered too.

The problem it solves

Evals are only as good as their test cases. Too few, too easy, or not like real usage, and your scores look great while users suffer. Cases that change every week make scores impossible to compare. And for rare-but-important situations (abuse, edge cases, other languages), you may have almost no real examples.

How it works: golden datasets

Collect real inputs: sample from production logs (anonymised), support tickets, user research. Aim for the real mix of questions.
Add deliberate cases: known hard questions, edge cases, past bugs, unanswerable questions, adversarial inputs, other languages.
Write the expected result for each case: an exact answer, required facts ("mentions 25 days"), required behaviour ("refuses", "asks for the order number"), or the relevant documents for retrieval.
Tag every case: category, difficulty, source, language, so you can slice scores.
Review it: domain experts check expected answers; disagreements get resolved and the guideline improved.
Version it (in Git or a dataset tool) so every score names the dataset version it used.
Split it: a development set you iterate against, and a held-out test set you check rarely (overfitting).
Grow it: every production failure becomes a new case; retire cases that no longer reflect the product.

How it works: synthetic test data

Use an LLM to generate cases where real ones are scarce: paraphrases of real questions, typos and slang, different languages, edge cases ("ask about carrying over 8 days when the limit is 5"), adversarial prompts, questions per document section.
Seed generation with real examples and your documents so cases stay realistic.
Generate expected answers too (or let a human write them) and review before adding: synthetic cases can be unrealistic, wrong or too easy.
Tag them as synthetic so you can compare scores on real vs synthetic cases.
Never rely only on synthetic data: models generate the kinds of questions models find easy.

Source	Strength	Weakness
Production logs	Realistic mix	Few rare cases; privacy work needed
Expert-written cases	Hard, important scenarios	Slow, biased by what experts imagine
Synthetic (LLM-generated)	Fast coverage, variations, rare cases	Can be unrealistic or wrong; must be reviewed

Where golden cases come from, and how the set keeps growing.

See it in code

Needs: Nothing: plain Python 3. Variations come from templates here; in practice an LLM writes them, seeded with real cases.

A small golden set with tags, plus synthetic variations generated from real cases.

golden_and_synthetic.py

import json
import random
from collections import Counter
 
# A golden dataset: real inputs, expected outcomes, and tags to slice results by.
golden = [
    {"id": "g1", "input": "How many days of leave do I get?", "expected": "25 days", "tags": ["leave", "easy"]},
    {"id": "g2", "input": "Can I carry over 8 days?", "expected": "max 5 days", "tags": ["leave", "edge"]},
    {"id": "g3", "input": "What's the meal limit in Paris?", "expected": "60 GBP", "tags": ["expenses"]},
    {"id": "g4", "input": "What's the CEO's salary?", "expected": "REFUSE_OR_UNKNOWN", "tags": ["unanswerable"]},
]
 
# Synthetic test data: generate variations of real cases (an LLM does this in practice; templates here).
random.seed(7)
STYLES = ["{q}", "hey quick q: {q}", "{q} thanks!!", "{q_lower}"]
synthetic = []
for case in golden[:2]:
    for style in random.sample(STYLES, 2):
        q = style.format(q=case["input"], q_lower=case["input"].lower())
        synthetic.append({**case, "id": case["id"] + "-syn", "input": q, "tags": case["tags"] + ["synthetic"]})
 
print("golden cases   :", len(golden), dict(Counter(t for c in golden for t in c["tags"])))
print("synthetic cases:", len(synthetic))
for s in synthetic:
    print("  ", json.dumps(s["input"]), "->", s["expected"])

Output

golden cases   : 4 {'leave': 2, 'easy': 1, 'edge': 1, 'expenses': 1, 'unanswerable': 1}
synthetic cases: 4
   "How many days of leave do I get? thanks!!" -> 25 days
   "How many days of leave do I get?" -> 25 days
   "can i carry over 8 days?" -> max 5 days
   "Can I carry over 8 days? thanks!!" -> max 5 days

Note one synthetic "variation" is identical to the original question: exactly the kind of low-value case a reviewer should drop. Store datasets as JSONL in Git, or in an eval tool's dataset feature with versions.

A real example

Example

A legal assistant's golden set

A legal-tech company built its golden set in three passes. First, 200 real questions sampled from beta users, each answered and checked by a lawyer. Second, 60 cases from known pitfalls: jurisdiction differences, superseded laws, questions that must be refused. Third, 300 synthetic paraphrases and multilingual versions generated by an LLM, of which reviewers kept about 180.

Scores are always reported per slice. When a model change raised the overall score but dropped "jurisdiction" cases by 15 points, the slice caught it. Every reported customer problem now becomes a new case within a day.

The easy way

Start here

Start a golden.jsonl file with 30 real questions: what was asked, what a good answer must contain, and one tag each. Add 5 unanswerable questions. Grow it by one case every time something goes wrong in production.

When you grow: the production setup

Teams manage datasets in eval platforms or versioned storage, with annotation guidelines, multiple reviewers and agreement checks (human labeling), PII scrubbing, stratified sampling from production, synthetic generation pipelines with review, dev and held-out splits, and dashboards showing scores per slice over time.

Trade-offs

What you gain

A stable, trusted yardstick for every change.
Slices show exactly where quality changes.
Synthetic data fills gaps cheaply, especially rare and adversarial cases.

What it costs you

Building and reviewing good cases takes expert time.
Datasets go stale as the product and users change.
Synthetic cases can be unrealistic and inflate scores.

When not to use it

Don't use a public benchmark as your golden set; it rarely reflects your users or tasks.

Common mistakes

Only easy, happy-path cases.
No unanswerable or adversarial cases.
Relying entirely on unreviewed synthetic data.
Tuning prompts on the same cases you report as your final score.
Copying real user data without removing personal information.

Interview questions

Try answering out loud first, then open the model answer.

How do you build a golden dataset for an LLM feature?

Model answer

Sample real, anonymised inputs representing the production mix; add expert-chosen hard cases, past bugs, unanswerable and adversarial cases; define expected outputs or properties with domain experts following written guidelines; tag by category and difficulty; review with multiple annotators; version it; split into dev and held-out sets; and continuously add production failures while pruning stale cases.

What are the benefits and risks of synthetic test data?

Model answer

Benefits: cheap coverage of paraphrases, languages, rare scenarios and adversarial inputs; per-document question generation for RAG; quick bootstrapping. Risks: unrealistic or trivially easy cases, wrong expected answers, model-specific biases that inflate scores, and leakage if generated by the same model being tested. Mitigate with seeding from real data, human review, tagging and reporting real versus synthetic slices separately.

How do you avoid overfitting your system to your eval set?

Model answer

Keep a held-out test split that you don't iterate against, refresh the dataset with new production cases, use many diverse cases rather than a few, track performance on online metrics too, and be suspicious of large gains on the dev set that don't show up on held-out data or in production.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Golden dataset: A trusted, versioned set of test inputs with expected outputs or labels, used to measure an AI system repeatably.
Synthetic test data: Test cases generated by an LLM (variations, edge cases, rare situations) to extend a golden dataset. Needs human review.
Eval: A repeatable test of an AI system's quality on a set of examples with a scoring method.
Data contamination: When test questions appeared in a model's training data, making its benchmark score look better than its real ability.

Before and after

Read first

065 Why evals matter