Golden dataset

A golden dataset is the small, hand-curated set of rows you trust as your source of truth. It's the dataset your release gate points at, the one you use to compare two models fairly, and the one you never mutate casually.

The golden set is not a bigger dataset that got smaller. It's a different kind of dataset. Every row was written or approved by someone who understands the domain, every row exists for a specific reason, and every row earns its place because a real failure or a canonical success is behind it.

If you only maintain one dataset per project, it should be this one.

Why golden datasets exist

Three jobs that no other dataset does as well:

1. Release-gate signal. The pass rate on the golden set is a number you can put in the release checklist. If it drops below the threshold, you don't ship. Living datasets grow and shift, their pass rate isn't comparable release to release; a stable golden set is.
2. Model comparison anchor. When you swap the target model, the only fair comparison is against a dataset that stays constant. The golden set is that constant.
3. Regression alarm. Every failure a real user or your team ever found becomes a row here. The next time that class of failure surfaces, the golden set catches it before production does.

Living / exploratory datasets are for coverage: the long tail, the edge cases, the "what haven't we tested yet". Golden datasets are for regression: the "we've decided this must always work".

Don't mix the two. See the Golden vs living datasets section on the hub for the split.

What a golden row looks like

A row belongs in the golden set only if it carries all four of these:

1. A specific, canonical input. Not a vague scenario; the exact phrasing the user (or a class of users) would produce.
2. An expected outcome or rubric. Either a gold expected output (for supervised metrics like Answer Precision) or a Custom Eval rubric that describes the criteria for pass. Or both.
3. Metadata that survives without you. At minimum category / tag, difficulty, and a source field (hand, production, ticket, incident). Anyone who reads the row six months from now should be able to tell why it's there.
4. A reason to exist. Written in one sentence, either in a dedicated note column or in the row's description. "Refund flow for a damaged item, order id in message body." "The bug from incident #142 where the agent quoted the wrong currency." Rows without a reason turn into noise once the person who added them leaves.

Rows that don't pass all four checks belong in a living dataset, not here.

Where golden rows come from

A mature golden set has rows from every one of the following sources. The mix matters more than the count.

1. Hand-written by a domain expert

The gold standard, no pun intended. Someone who knows the product sits down and writes a set of canonical scenarios: happy path, plausible edge cases, must-refuse scenarios, forbidden-topic tests.

• When to do this. Every project starts here. Write 20-50 rows by hand before you touch AI generation.
• Time per row. 3-10 minutes for a good row. A day of focused work produces 30-60 rows.
• What to write first. Canonical happy-path scenarios for every documented business scenario in the project. Then out-of-scope and forbidden-topic rows for every restriction. Then edge cases you personally know exist.
• What to skip. Don't try to invent robustness / metamorphic / synthetic-attack rows by hand. Those are what generation is for.

2. Promoted from production traces

The most valuable source of golden rows once the SDK is streaming traffic. Real user turns are more diverse, more surprising, and more important than anything you'd write.

The promotion workflow:

1. Once the runtime tracing SDK is wired into your production agent, sessions appear in the trace view.
2. Filter traces to interesting ones: low-confidence metrics on the auto-eval side, high latency, thumbs-down user feedback, tickets the customer support team escalated, anything you'd rather never see again.
3. Open the trace. Read the transcript. Decide: does this represent a class of interaction the product must handle?
4. If yes, click Add to dataset on the trace to promote it into a golden dataset. Add the metadata: category, difficulty, source = production, note explaining what the row represents.
5. Author the expected outcome if you can. Traces come without a gold answer; you have to add one for supervised metrics to score the row.

Cadence. Weekly review of 10-30 traces. This is where the golden set grows steadily and stays relevant.

3. Extracted from support tickets, user feedback, incidents

Every user complaint that reaches your support team is a candidate row. Every "the AI told me X and it was wrong" ticket is a row you never want to fail on again.

• Support tickets: turn the failing user message into an input, the correct answer into expected_output, tag it with the ticket id in metadata.
• Postmortems: after any incident where the agent behaved badly, add rows that reproduce the bug. Even after the fix ships, the rows stay as a permanent regression alarm.
• Escalations: any question the agent couldn't handle and had to hand off to a human is a signal. If the human's answer was correct, the pair becomes a golden row.

Cadence. After every sprint retrospective or every incident postmortem, add the rows that came out of it. This is how the golden set grows through the org, not just through the eval team.

4. Curated from AI-generated rows

The generator produces rows quickly. Most of them are OK. A few are excellent, and those excellent ones deserve promotion to the golden set.

• Generate a batch (in a living dataset).
• Run an eval and read the failures + borderline passes.
• The rows that reveal real product weaknesses are the ones worth keeping.
• Copy those rows into the golden set. Edit them: tighten the input phrasing, write a canonical expected output, add metadata.
• Discard the rest.

The mindset shift: generated rows are candidates, not members. Promotion is a judgement call, not an automation.

5. Migrated from your legacy test bank

If you had a test suite before EvaliQA (a spreadsheet of Q&A, a Postman collection, a set of prompts your PM would try after every model change), most of those rows belong in the golden set.

• Import via CSV / JSONL to a fresh dataset.
• Manually review: which rows still reflect the current product scope? Which are stale?
• Trim ruthlessly. A legacy 500-row spreadsheet usually becomes a 60-row golden set.

6. Regression rows from every bug fix

The rule: every bug that shipped to production becomes a permanent row in the golden set. After the fix, the row stays. The pass rate on this subset should stay at 100% forever; any drop means the same bug came back.

Tag these rows with source = incident and the incident id. The subset of source = incident rows is your regression checklist.

How to seed a golden set from scratch

Two weeks of work, roughly:

1. Week 1, Monday. Write out your project's business scenarios in one column of a spreadsheet: refund, escalation, order status, product question, complaint, etc. Aim for 8-15 scenarios.
2. Week 1, Tue-Thu. Hand-write 2-4 canonical happy-path rows per scenario. Add expected outputs. That's 20-40 rows.
3. Week 1, Fri. Add out-of-scope / forbidden-topic rows for every restriction. Add 2-3 per restriction. That's another 5-15 rows.
4. Week 2, Mon-Tue. Wire in production traces (if the SDK is deployed) and promote the 10 most interesting real conversations into golden rows.
5. Week 2, Wed. Run the plan against the golden set for the first time. Read every failure. Some are product bugs, some are dataset bugs (bad expected output, wrong metric). Fix both.
6. Week 2, Thu-Fri. Iterate. Aim for 60-80 rows total with a clean run.

You now have a golden set. From this point it grows by 1-5 rows a week from real signals.

Sizing

A good golden set is small on purpose:

• 10-50 rows for a new project. Enough to spot most regressions, fast to run.
• 50-150 rows for a mature project. Broad coverage of your business scenarios and known failure classes.
• 150-300 rows for large / regulated products where compliance demands breadth.

Above 300 rows the dataset starts blurring into a living dataset, harder to keep every row justified. Split into a smaller "golden core" and a bigger "living regression" set if you outgrow this.

Cost per run scales linearly. A 100-row golden set with 3 LLM metrics on a strong judge costs $2-10 per run on GPT-4-class models. Design for weekly / nightly / per-PR cadence and check your monthly budget.

Sources that should NOT go into the golden set

Everything above is what belongs. Everything below does not:

• Bulk-generated rows without review. Generated content is a living-dataset thing. Anything in golden was reviewed and approved.
• Trivial "hello / hi / thanks" chit-chat. Not a canonical scenario. Move to a smoke-test dataset if you must test it.
• Rows that duplicate existing ones with a slight tweak. Duplication inflates pass rate artificially. Use metamorphic testing in a separate dataset for phrase-robustness.
• Rows you're unsure about. If you don't know why a row is in the set, next month you won't either. Delete.
• Rows for retired features. Every quarter, prune rows that test scope the product no longer serves.
• Trace snapshots without editing. A raw production trace is a starting point, not a finished row. Add the expected outcome, the metadata, and the note before promoting.

How to use it

Once the golden set is in place, wire it in:

• Release gate. Create a test plan that runs against the golden set. Point CI at it. If pass rate drops below your threshold on the PR that triggered the run, block the merge.
• Model comparison. Whenever you swap the target model, run the current model against the golden set, then run the candidate model against the same golden set. Compare in the test plan's Eval runs tab.
• Nightly heartbeat. A nightly run against the golden set catches slow regressions from external drift (provider model updates, retrieval index changes) that per-PR runs miss.
• Baseline for the report. Every report should quote the golden-set pass rate as its headline number.

See Evaluation pipeline for how the golden set fits into the broader CI / nightly / weekly cadence.

Reviewing and pruning

A golden set that only grows eventually turns into archaeology. Every quarter, do a review pass:

• Read every row that hasn't been touched in 6+ months. Does the row still reflect the current product? If not, edit or delete.
• Remove rows for features that shipped and are no longer supported. Product scope changes; the golden set should follow.
• Merge near-duplicates. If two rows test the same thing with slightly different phrasings, keep one and add the other to a metamorphic dataset.
• Refresh the reasons. If a row's note column says "for the incident from 2024", check whether that incident's fix is still in place. If yes, update the note; if it was reverted, the row is still doing its job.

Common mistakes

• Generating a "golden set" via AI without curation. The result is a living dataset with a fancy name. Golden means hand-reviewed.
• Growing without pruning. After 500 rows you can no longer reason about the set. Trim aggressively.
• Copying production traces verbatim. Traces come without expected outputs and without categorisation. Editing is required before promotion.
• Skipping the metadata. A row without category and source is a row you can't slice on. Add them at promotion time, not later.
• Never running the golden set. A dataset that isn't in the release gate isn't actually golden. Make it the CI-blocker.
• Editing rows in place after they've been used as baselines. Old runs still point at the version they used, but "the golden set at pass rate 91%" stops meaning what it did if the rows shifted. Use Duplicate as new version for big edits, keep the frozen version linked to the old runs.
• Deleting incident-source rows because the bug was fixed. Never. That's the entire point of adding them. The bug can come back, from a model update, a prompt tweak, a dependency bump. The row is the alarm.

Related

• Datasets, methodology hub
• Evaluation dataset types
• Evaluation pipeline: how the golden set fits into the broader CI / nightly cadence.
• Evaluation results: reading the runs the golden set produces.
• Runtime tracing SDK setup: the source of promoted rows.