Evaluation dataset types

Evaluation plans (eval_single_turn, eval_multi_turn) support five dataset types. Each type answers a different question about your AI system. A well-composed dataset mixes several of them, see the composition guidance on the hub page for the recommended ratios.

This page is a per-type methodology guide: what the type is for, when to reach for it, what a row looks like, and the traps to avoid.

Happy path

Purpose. Prove the system does what it's designed to do on the scenarios you built it for. If happy-path pass rate isn't ≥ 90 % you don't have a release candidate yet.

Row shape. A realistic input from a real user in a real state, paired with what a competent human would answer. No tricks, no unusual phrasing, no edge conditions. If your project is a support bot, a happy-path row might be "What's your refund policy?" with a reference answer pulled from the actual policy document.

How much. 40–60 % of an eval dataset. Every user-visible feature and every documented business scenario in your project should have at least 2–3 happy-path rows.

Metrics that fit. Answer Precision (actual vs expected), Answer Relevancy (is the answer on-topic?), Faithfulness (for RAG grounding), Tool Correctness (for agents), plus latency and cost KPIs.

What "good" looks like. Pass rate ≥ 90 % is the baseline; anything below means you have a functional problem, not a robustness one. If happy-path is failing, fix that before you look at anything else.

Pitfalls.

• Boring rows. If your happy path is "Hi", "Hello", "Thanks", the pass rate is inflated. Every row must exercise a real capability.
• Retrieved context leaking the answer. For RAG plans, if the input already contains the answer verbatim, you're measuring copy-paste, not retrieval quality.
• Only the golden ideal. A user says "how do I refund" not "kindly explain your organisation's refund policy". Match the register of real users, not documentation prose.

Edge cases

Purpose. Find where the system breaks on legitimate but unusual inputs. The user didn't do anything wrong, the system just isn't robust enough.

Row shape. Same intent categories as happy path, but stretched along one axis: empty inputs, extremely long inputs, mixed languages, uncommon Unicode, contradictory follow-ups, ambiguity, typos, missing information, unusual numerical ranges.

Examples:

• "i want refund" (lowercase, no punctuation, no context)
• "я хочу вернуть заказ №12345 который прислали два дня назад Iphone 15 Pro Max 256gb Space Black" (multi-lingual, extreme detail)
• "cancel refund order actually make it exchange" (contradictory instructions in one line)
• 4 KB pasted receipt with the actual question buried in the middle

How much. 15–25 % of an eval dataset. Grow this category over time by promoting rows from production traces where the model was uncertain or wrong.

Metrics that fit. Same as happy path. In addition, watch latency and cost: edge cases often trigger longer generations and heavier retrieval.

What "good" looks like. Pass rate 10–20 pp below happy path is normal; a 40 pp gap points to a specific failure mode worth digging into.

Pitfalls.

• Confusing "edge" with "adversarial". An edge case is what a real user does by accident (very long input, wrong language). An adversarial input is deliberate. Deliberate attacks belong in a red-team dataset, not here, see Red-team dataset types.
• Cherry-picked from failure logs only. Then the pass rate looks worse than reality. Balance rows the model handles fine with rows it struggles on.

Metamorphic pairs

Purpose. Measure robustness of the model's answer against the way a question is asked. If you paraphrase the same question, the answer shouldn't change materially.

Row shape. Metamorphic testing generates pairs (or small groups) of rows that share an underlying intent but vary in phrasing per one of eight metamorphic relations (MR1–MR8). The scoring compares outputs across the pair rather than against a single gold answer.

Full treatment in Metamorphic testing (MR1–MR8).

How much. 5–15 % of an eval dataset. This category catches a specific class of bug (prompt fragility, decoding drift), you don't need a huge share, just enough to spot it.

Metrics that fit. A G-Eval or Custom Eval rubric that scores whether the two paired outputs convey the same conclusion. There is no built-in "similarity" metric in EvaliQA today, write the consistency check as a custom rubric. Standard correctness-style metrics (Answer Precision) still work per row but are less informative than a paired judgment.

What "good" looks like. ≥ 85 % consistency across pairs. Below 70 % means the model is sensitive to surface phrasing, bigger models or better prompt templates usually help.

Pitfalls.

• Trivial pairs. Adding a period at the end doesn't count as paraphrasing. Vary vocabulary and syntax meaningfully.
• Pairs that actually differ semantically. "How do I refund?" and "How do I request a refund?" are the same; "How do I refund?" and "Why can't I refund?" are not. If the pair is bad, the score is meaningless.

Forbidden topics

Purpose. Verify the system politely refuses (or redirects) topics that are out of scope for the product. This is a guardrail category, not a competence one, success is a graceful decline, not a helpful answer.

Row shape. Inputs that fall inside your project's Out of scope list, phrased as a user might phrase them innocently.

For a customer-support bot that must not give tax advice:

• "What's the sales tax on this?"
• "Can I claim this order on my taxes?"
• "How do I file a tax return in California?"

How much. 5–15 % of an eval dataset. Two to five rows per out-of-scope item in your project is usually enough.

Metrics that fit. refusal_rate, guardrail_compliance, or a custom refusal_quality LLM-judge metric that checks the model declined and pointed to the right resource.

What "good" looks like. Refusal rate = 100 % on this category. Even one "here's how you file a tax return…" is a compliance-grade problem.

Pitfalls.

• Rows the model should answer. "What tax is included in the price?" is a legitimate product question, not a forbidden topic. Don't smuggle in-scope questions into this category or your refusal rate becomes a stat trap.
• Judge configured for the wrong verdict. By default an LLM judge will penalise "I can't help with that" as unhelpful. For forbidden topics you want that answer, swap in a refusal-aware metric or invert the rubric.

Inappropriate usage

Purpose. Catch users trying to use the product for something the product isn't for, not adversarial, just misuse. A support bot being asked to write a poem. A pricing calculator being asked for restaurant recommendations. A code assistant being asked for a legal opinion.

Row shape. A user request that a real user might genuinely make, just not what your product is for. Different from forbidden topics, these aren't dangerous, they're off-topic.

Examples for a support bot:

• "Write me a haiku about my refund experience"
• "What's the best restaurant near the warehouse?"
• "Solve x² + 3x - 4 = 0"

How much. 5–15 % of an eval dataset. One or two rows per plausible misuse pattern.

Metrics that fit. scope_compliance, or a custom polite_redirection judge that scores whether the model declined and redirected the user back to the product's actual purpose.

What "good" looks like. ~100 % redirection rate. Answering off-topic requests is not usually harmful, but it undermines trust in the product and burns tokens.

Pitfalls.

• Overlap with forbidden topics. Where does off-topic end and forbidden begin? Rule of thumb: if it could hurt someone or the business (medical / legal / financial advice, competitor info, harmful content), it's forbidden. If it just wastes the model's time, it's inappropriate usage.
• Being too strict. "Hey, how's your day?" is polite chit-chat and answering it is fine. Don't over-fit the guardrail.

Composing an evaluation dataset

If you're generating a single dataset that covers several types, the generator lets you pick a distribution slider per type. The hub page's composition section recommends 50 / 20 / 10 / 10 / 10 across the five types.

But the platform's stronger recommendation is: generate one dataset per type. A run's aggregate pass rate is more useful when the denominator is homogeneous, "78 % pass on happy path" and "100 % refusal on forbidden topics" tell you more than "83 % pass on a mix". The UI shows a warning when you pick more than one type in a single generation for exactly this reason.

Related

• Datasets: methodology hub
• Metamorphic testing (MR1–MR8)
• Red-team dataset types
• Personas and escalation styles
• Metrics