Test plans

A test plan is the recipe for one evaluation. It answers "what are we testing, how do we call the system, what does 'good' mean, and where do the test cases come from?", bundled together so you can reuse it across runs, compare runs against each other, and put the same plan in CI.

If a dataset is what you evaluate against, a test plan is how you evaluate. The two are decoupled on purpose: one plan can run on many dataset versions over time, and one dataset can be attached to several plans.

This page is the methodology of building good test plans. For the mechanics of each mode, what a row looks like, which metrics fit, and how to size the run, use the sub-pages below.

Anatomy of a test plan

Every plan has five things bound to it:

1. Channel: how EvaliQA reaches the system under test.
   • Text: HTTP connectors (REST, GraphQL, webhook).
   • Voice: EvaliQA's AI places phone calls, records audio and transcript.
2. Mode: the shape of the interaction. Four modes total; see the next section.
3. Model + judge model: one LLM credential and model pair for the run's own calls, one for the LLM-as-a-judge (may be the same). The judge is plan-wide, not per-metric.
4. Metrics: how each row is scored. Built-in from the catalog plus your workspace's custom metrics. Metrics carry parameters and thresholds.
5. Datasets: one or more attached datasets that feed rows to the run. Only datasets whose template matches the plan's mode can be attached.

Plans are workspace-scoped and always attached to a project: you can't create a plan without picking one.

The four modes

EvaliQA has exactly four test plan modes. Two evaluation, two red-team. Multi-turn evaluation has three sub-strategies you'll pick during setup.

Mode	Purpose	Sub-page
eval_single_turn	One-shot Q&A, the simplest and cheapest way to measure quality.	Evaluation: single-turn
eval_multi_turn	Conversations, an AI user or a script talks to your agent for N turns. Three strategies: Simulation, Scripted, Adaptive.	Evaluation: multi-turn
redteam_single_turn	Adversarial single prompts, prompt injection, jailbreaks, PII, harmful content.	Red teaming: single-turn
redteam_multi_turn	Adversarial dialogues, crescendo, linear jailbreak, persistent context, stealth.	Red teaming: multi-turn

Voice is a channel, not a separate mode. Any of the four modes can run over voice if your workspace has voice eval enabled, the plan just uses phone-call + STT + TTS instead of HTTP.

How to pick a mode

Start narrow, expand deliberately. A typical maturation path:

1. Start with eval_single_turn. Every new project. Fast, cheap, forces you to define ground truth. If you can't score your system on 30 one-shot Q&A rows, you're not ready for anything harder.
2. Add redteam_single_turn before you launch to real users. Any customer-facing AI product needs this as a release gate. It's a different question from correctness, even a highly-scoring eval_single_turn plan can leak PII or take an authority attack.
3. Add eval_multi_turn when the product is inherently conversational, support chat, voice assistants, agents that maintain state. Skip it for stateless single-shot products.
4. Add redteam_multi_turn for products with a real attack surface, anything with an API, anything handling sensitive data, anything where a user could plausibly try to jailbreak over several turns.

A mature product has all four running on different cadences: single- turn on every PR, multi-turn nightly, red-team weekly to monthly.

Judge model: pick it once, keep it fixed

The judge model is the LLM you use for LLM-as-a-judge metrics (correctness, faithfulness, relevancy, and most custom metrics). On platform credits it is the platform model, chosen for you; when the Platform AI agent runs on your own credential it's set on Step 3 of the wizard and applies plan-wide.

Two rules that save you pain later:

• Pick a strong judge, once. GPT-4-class or Claude-4-class as the judge is the sweet spot: reliable enough to give reproducible verdicts, cheap enough at judge-only-once-per-row. Weaker judges produce noisier scores.
• Don't change the judge model between runs you want to compare. Judge model is part of the measurement instrument. Swap it and your historical numbers stop being comparable.

If cost is a concern, use a cheap model for the system under test and a stronger model for the judge. That combo gives the best signal-to-cost ratio.

Metrics on a plan

Metrics attach at the plan level, not the dataset level, that way one dataset can be scored differently by different plans.

Two flavours:

• Built-in: the eval-ai-library catalog: correctness, relevancy, faithfulness, hallucination, PII leakage, toxicity, latency, cost, ~30 more. See the Metrics concept.
• Custom: presets you create in /custom-metrics and reuse across plans. Custom metrics can be CustomEvalMetric (numeric score with a threshold) or GEval (categorical labels).

Each metric carries parameters (rendered on wizard Step 5) and a threshold. The threshold turns a raw score into pass / fail: a metric with threshold: 0.7 marks a row as passed iff the score is ≥ 0.7.

Red-team modes skip the metrics step entirely. The runtime scores each row automatically per (vulnerability, attack technique), the metric is baked into the dataset. See Red-team dataset types.

Plan status: the lifecycle

Every plan has a status you can flip from the header:

• Draft: you're still editing. Runs are allowed: draft is a workflow label, not a gate. Use it to signal "don't rely on this yet".
• Ready: the plan is validated and safe to compare across runs. This is what you point CI at.
• Archived: hidden from the default list. Historical runs still work; you just don't want the plan cluttering the list any more.

The button in the header flips between these. There's no auto- transition; it's a human signal.

The plan × dataset relationship

• One plan can have many datasets attached (typically different versions of the same golden set, or one per dataset type).
• One dataset can be attached to many plans (via Attach existing , the rows are cloned as a new version under the target plan).
• A dataset's template must match the plan's mode. You can't feed a redteam_single_turn dataset to an eval_multi_turn plan.

When you press Run eval, the sheet asks which dataset version to run against (only shown if the plan has more than one). Each run is tagged with the exact dataset id it used, so runs stay reproducible even after you version the dataset.

Iteration cadence

A test plan is not a one-time artefact. Reasonable rhythms once you have the plan running:

• Every code change: run the smallest golden eval_single_turn plan (20–50 rows). Should complete in under a minute; block the merge on regressions.
• Nightly: run the full eval_single_turn plan plus a small eval_multi_turn set. Look for drift, not urgent breaks.
• Weekly: run redteam_single_turn on the current model / prompt. Prompt injection defence drifts, and provider updates change the baseline.
• Monthly: run redteam_multi_turn plus a big eval_multi_turn sweep. Refresh the golden set from production traces.
• Every model change: full pass on all four modes plus a run-to- run compare against the previous baseline. Model swaps are the highest-risk change you can make.

Cost: the shape of what you pay

A plan doesn't cost anything sitting there. Cost accrues when you run it. Rough shape per run:

cost ≈ rows
     × (target_LLM_calls + judge_LLM_calls_per_metric × #LLM_metrics)
     × (avg_input_tokens + avg_output_tokens) × provider_price

Concretely:

• eval_single_turn: 1 target call + 1 judge call per LLM metric. A 100-row run with 2 LLM metrics is 300 model calls.
• eval_multi_turn: N turns × (1 target + 1 simulator/adaptive agent call) + judge calls at the end. A 20-conversation × 5-turn run is easily 200+ model calls.
• redteam_single_turn: 1 target call + 1 auto-judge call per row. Attacker LLM (if configured) adds one more per row.
• redteam_multi_turn: the heaviest. Every turn is target + attacker + eventual judge. Attacks can be dozens of turns each.

The KPI tile Total cost on the run detail page tells you the retro number in USD to four decimals. Sanity-check it after your first run of each plan, if it's an order of magnitude larger than you expected, revisit dataset size, turn count, and metric count.

Tips and pitfalls

• One plan per mode per project. Don't try to fit red-team and correctness into one plan. Different questions, different metrics, different cadences.
• Freeze the judge model. Compare across runs only when the judge is identical.
• Name plans by what they test, not what they run against. Support-bot correctness, golden 200 is better than Prod plan v3.
• Small before big. Prove the plan works on 20 rows before you generate 500. A misconfigured plan on 500 rows is a $50 mistake.
• Attach a connector before running. The Run eval sheet asks for one; a plan without a connector can't call anything. Set connectors up on the project's Connectors tab.
• Watch the Adaptive multi-turn gate. Adaptive strategy needs the Platform AI agent to be configured, or the option is greyed out on the mode step.
• The Documents step is single-turn only. Attaching reference files works for eval_single_turn generation. Multi-turn plans ignore documents even if they're uploaded.

Dive deeper

Each mode has its own methodology page, what a row looks like, which metrics fit, how many rows is enough, what "good" means, and the pitfalls specific to that mode:

• Evaluation: single-turn
• Evaluation: multi-turn (Simulation, Scripted, Adaptive)
• Red teaming: single-turn
• Red teaming: multi-turn

Related

• Run your first evaluation
• Datasets
• Metrics
• Custom metrics