Datasets

A dataset is the ground truth of an evaluation. Every row is one test case: an input to the AI system, ideally an expected output or a rubric, and any context the system needs to produce a fair answer. When you run a test plan, the eval engine iterates over the dataset row by row, scores each result against the metrics, and returns a verdict.

Everything else in EvaliQA gets easier or harder based on the quality of your datasets. A great model on a bad dataset scores worse than a mediocre model on a well-curated one. This page is the methodology, how to think about datasets, what to put in them, and how to grow them so evaluations stay comparable over time.

If you want the UI reference for editing rows, see the Datasets page. If you want to know the mechanics of a specific dataset type, use the sub-pages below.

The three things a dataset must give the eval engine

Whatever the plan mode, every dataset needs to answer three questions per row:

1. What goes in: the user input (a question, a chat turn, a document to summarise). For multi-turn plans this is a persona and a scenario, not a single string.
2. What "good" looks like: an expected output (for Answer Precision or Contextual Recall), a reference document (for Faithfulness), a set of forbidden phrases (for guardrails), or a rubric the LLM judge follows. Not every metric needs a gold answer, but every metric needs some notion of ground truth.
3. Anything the system needs to do the job: retrieval context, tool catalogue, system prompt, previous messages. If the metric wants to check faithfulness against passages, the passages must live on the row.

If any of the three is missing, either the metric can't run or the run produces noise. Design the dataset to satisfy the metrics you'll attach, not the other way around.

Golden vs living datasets

Two archetypes, both useful. Most projects end up with one of each.

Golden datasets are hand-curated, small (10–200 rows), and change rarely. Every row was written or approved by someone who knows the domain. The goal is a regression alarm: the pass rate on the golden set is your release-gate number. If it drops below 90 %, you don't ship. Golden sets are also what you use to compare two models fairly, because they're the one axis you keep constant across runs.

Living datasets are large (hundreds to thousands of rows), grow with production traffic, and drift on purpose. They give you coverage: the long tail of edge cases, the weird phrasings, the multilingual outliers. You update them weekly or nightly by promoting interesting production traces (via the trace view) and by generating synthetic rows for gaps you notice.

You compare models on the golden set. You explore weaknesses on the living set. Don't mix the two, pass-rate deltas on a moving target aren't meaningful.

Where rows come from

EvaliQA lets you build datasets four ways, and a mature project mixes all four.

1. Hand-written by a domain expert

The gold standard for the golden set. Slow (a good row takes 3–10 minutes if you have to think about the expected answer), but every row earns its place. Do this for the first 20–50 rows so you know what a right answer looks like before you scale.

2. AI-generated from a project description

The Generate action in a test plan writes seed rows from your project's Description, Business scenarios, Capabilities, and Out-of-scope lists, plus, optionally, uploaded reference documents. Great for a first pass and for filling coverage gaps once you know what you want.

Two things to know before you generate:

• The quality of what comes back is a direct function of the project's Context / Scope fields. A thin project description produces generic rows. Fill in the fields before you generate.
• Always review the first batch. The AI does not know your business the way you do. Delete rows that miss the point, edit the ones that almost work, promote the good ones to the golden set.

3. Promoted from production traces

Once the SDK is streaming traces from your live agent, you can pick real conversations from the trace view and promote them into a dataset with one click. This is how living datasets grow: every failure a real user found becomes a row you'll never regress on again.

4. Imported from CSV / JSONL

Have an existing test bank in a spreadsheet, a Postman collection, or a CI pipeline? Upload it as CSV or JSONL. Column names in the file map to the dataset schema, see Datasets page for the exact mapping rules.

What to include in each row

Beyond the input and expected output, add whatever helps you slice results later. EvaliQA row-level metadata columns are cheap and pay for themselves the first time you need to filter by them.

Recommended metadata:

• category or tag: a short label for what this row is testing (billing.refund, support.escalation, edge_case.empty_input). Every chart, filter, and report can group by this.
• difficulty: easy / medium / hard. Lets you tell "we passed everything easy" from "we passed everything except the tricky ones".
• persona: for multi-turn plans especially: expert, novice, impatient, non-native speaker. See Personas and escalation styles.
• source: hand, generated, production, import. Useful when auditing coverage and figuring out which rows to prune.
• created_at and last_reviewed: the reviewer's date-of-record. A row that hasn't been sanity-checked in a year probably needs a look.

Only add fields you'll actually use. Extra columns cost nothing to store but do clutter the editor.

Dataset composition: the mix that matters

Real coverage comes from a distribution of rows, not just a count. When you set up a dataset for an evaluation plan, aim for a rough mix like:

• ~50 % Happy path: the system doing what it's designed to do, well.
• ~20 % Edge cases: legitimate but unusual inputs (empty strings, extremely long, unicode, multi-lingual, ambiguous phrasings).
• ~10 % Metamorphic pairs: the same intent expressed several ways, to catch prompt fragility.
• ~10 % Forbidden topics: things the system should refuse or redirect.
• ~10 % Inappropriate usage: attempts to use the system for something it shouldn't do.

For a red-team plan the mix is different, see Red-team dataset types.

How many rows is enough?

There's no magic number, but there are ballparks that work well in practice:

• Smoke test / gate for CI: 20–50 rows on the golden set. Enough to spot most regressions, small enough to run in under a minute.
• Release qualification: 100–300 rows across a balanced mix. Enough to give a defensible pass rate.
• Coverage / exploration: 500–2000 rows, mostly generated + promoted from production. This is where you find long-tail failures.
• Red-team pass: 3–5 attacks per vulnerability × the vulnerabilities you care about. See Red-team dataset types.

Going much bigger has diminishing returns and starts to cost real money, every row is at least one LLM call for the target, plus one per LLM-based metric. A 2 000-row plan with a GPT-4-class judge can be tens of dollars per run.

Analytics you should keep an eye on

EvaliQA doesn't yet ship a "dataset analytics" tab, but the numbers you want are visible on adjacent screens:

• Per-category pass rate. Compare the pass rate on billing.refund rows vs support.escalation rows in the run's per-row list, if one is 50 pp lower, that's where to look.
• Row age. In the dataset editor, sort by created_at. If half your rows were added more than six months ago, plan a review.
• Failure clustering. After a run, filter to Failed and read the judge's reasoning. If ten failures cite the same missing tool call, that's one bug, not ten.
• Cost per row. On the run detail, Total cost / Rows. If it creeps up over time, either the model got heavier or your rows got longer.
• Duplicate detection. Roughly: are two rows so similar the eval becomes double-counting? Currently a manual check in the editor, sort by input.

How to grow a dataset over time

A dataset is a living artifact, not a one-shot artefact. A reasonable cadence:

1. Every finished run: promote the interesting failures. Any row where the model was confidently wrong, or the judge's reasoning revealed a gap in the rubric, goes into a follow-up dataset for the next iteration. See "promote row to dataset" on the eval run detail.
2. Weekly: read 20 production traces (filter by low-confidence metric, high latency, or user thumbs-down). Promote the ones your golden set doesn't already cover.
3. Monthly: re-run the full plan on the golden set with the latest model version. Look for regressions.
4. Quarterly: review rows that haven't been touched in six months. Delete or update ones that reflect an old scope, prompt, or feature set. This is the pruning step; datasets that only grow eventually turn into archaeology.

Tips and pitfalls

• Design for the metrics you'll use. Adding Answer Precision or Contextual Recall after the fact to a dataset with no expected outputs is painful. Decide the metric mix before writing rows.
• Don't leak your expected outputs into the input. It sounds obvious; it happens. If your input includes the answer (via retrieved context or the row's phrasing), the score is not measuring what you think.
• One dataset, one purpose. A dataset mixing happy-path and red-team turns into "78 % pass", meaningless. Keep them separate.
• Version before mass edits. Use Duplicate as new version before a big change (bulk edit, mass regenerate). Old runs stay reproducible against the frozen version.
• Watch for hidden trivial rows. If half your happy-path rows are "hello" / "hi" / "hey", your happy-path pass rate is inflated. Deduplicate near-duplicates.
• Beware the language mix. If your project's Languages field has five entries, the generator will produce roughly equal amounts of each. If 95 % of your traffic is English, weight the dataset the same way.
• Guardrail rows need a "should refuse" verdict. A forbidden-topics row where the model says "I can't help with that" is a pass, not a fail. Metrics for those rows need to know that, usually via the row's expected_behavior = refuse or a dedicated guardrail metric.

Dive deeper

Each dataset type has its own mechanics, expected columns, and methodology traps. The sub-pages below go type by type:

• Evaluation dataset types, happy path, edge cases, metamorphic, forbidden topics, inappropriate usage.
• Red-team dataset types, vulnerabilities, attack techniques, escalation styles, how many rows per attack.
• Metamorphic testing (MR1–MR8), robustness pairs and how to read the results.
• Personas and escalation styles, the persona catalogue and when to use each in multi-turn plans.

Related

• Run your first evaluation, the practical walkthrough that connects a dataset to a plan and a run.
• Datasets page, UI reference for the editor, import / export, and inline row edits.
• Metrics, what each metric needs from a dataset row.