Evaluation, multi-turn

eval_multi_turn evaluates your agent as it holds a real conversation: several user turns, several agent turns, state carrying across, and a score at the end that judges the whole exchange, not one message.

This is where conversational products earn or lose their release. A chatbot that answers every one-shot question correctly can still fall apart when the user asks a follow-up, changes their mind, or comes back to a topic three turns later. Multi-turn evaluation is the mode that catches this.

Three sub-strategies live inside eval_multi_turn. They differ in who writes the user's turns and how much that user reacts to your agent, and the choice matters as much as the mode itself.

When to use eval_multi_turn

• Chatbots and support assistants: any product where users clarify, follow up, or reference earlier turns.
• Voice agents: every voice interaction is multi-turn by nature.
• Agents that gather info before acting: booking flows, KYC, troubleshooting flows.
• Anything where success is a goal completion, not a good answer: refund processed, appointment booked, ticket resolved.

Skip it for stateless products (a single-shot summarizer, a classifier, a search endpoint). Multi-turn on those is expensive theatre.

The three strategies

You pick the strategy on wizard Step 2, under the mode picker.

Strategy 1: Simulation (default)

What happens. The generator writes scenario seeds: a persona, an intent and success criteria (e.g. "impatient user, wants to refund a damaged item that arrived yesterday, order number is 12345"). At run time an AI user plays that persona against your system, one message per turn. It reads the whole conversation so far, including your agent's replies, so it answers a clarifying question and follows up the way the persona would. It ends the conversation when it has what it came for, or at max_turns.

Repeatable. The AI user runs at temperature 0 and has no tactics or plan goals to pursue, so the same seed against the same agent gives very nearly the same conversation each run. If your agent answers differently, the user reacts differently, which is the point.

Cost. One model call per user turn, on the same model as the plan's judge (platform credits or your own key). More than Scripted, less than Adaptive.

When to reach for it.

• Regression testing and CI, where you want realistic but stable conversations.
• Comparing two candidate models or prompts. The user side is fixed except for how it reacts to each candidate.
• Broad coverage from generated seeds without writing every turn by hand.

When it hurts. The AI user pursues the intent, not your test plan's goals. It won't probe for a specific weakness or change approach when the agent stalls. Use Adaptive for that.

Strategy 2: Scripted

What happens. The generator writes the entire user-side dialogue up front, turn 1, turn 2, turn 3, all pre-planned. The eval engine replays the script back verbatim.

Deterministic. Fully. No user-side LLM call at run time at all. Same script, same target model → same conversation.

Cost. The cheapest of the three by a wide margin. Zero user-side runtime LLM cost.

When to reach for it.

• You want to test specific dialogue paths: the exact turns a problem user might send, verbatim. Great for reproducing production incidents.
• You're testing a fixed IVR-like flow where the user side is effectively canned.
• You have a hand-written test bank of past support tickets you want to replay.

When it hurts. The user side can't react. If the agent asks "which order?" on turn 1, the scripted user still sends turn 2 as written. Great for regression, bad for measuring how the agent handles real dialogue. Simulation is the reacting alternative.

Strategy 3: Adaptive

What happens. Same scenario seeds as Simulation, but the user side runs on the Platform AI agent and works toward the plan's goals: each turn it notes what worked, tracks which success criteria are met, and chooses its next tactic.

Repeatable. Less than Simulation. The model also runs at temperature 0, but each turn it reads the plan's goals and its own notes from earlier turns and picks a tactic, so small differences in your agent's replies lead to different paths. Aggregate pass rates stay stable across runs; individual transcripts vary more.

Cost. The most expensive multi-turn strategy, you pay for the Platform AI agent's calls on every user turn on top of your agent's calls.

When to reach for it.

• You want the most realistic conversations. Adaptive users actually react to your agent, they ask follow-ups, they push back, they clarify.
• You're stress-testing an agent that only shines in real dialogue, where the value is handling deviations, not answering a script.
• You're exploring failure modes you haven't seen yet.

When it hurts. More variation makes tight regression testing harder. Adaptive needs the Platform AI agent: on credits it's ready with no setup; on your own key, pick a credential and model under Settings → Platform AI agent first, or the strategy is greyed out in the wizard.

The strategies side by side

	Scripted	Simulation	Adaptive
Who writes the user's turns	You or the generator, before the run (user_turns)	An AI user, live, from the persona and intent	The Platform AI agent, live, from the persona, intent and plan goals
Sees your agent's replies	No	Yes	Yes
Uses the plan's goals	No	No	Yes
Changes tactics during the dialogue	No	No	Yes, notes what worked and picks the next tactic
Model for the user side	None	The plan's judge model, temperature 0	The Platform AI agent, temperature 0
User-side cost	None	One call per user turn	One call per user turn, longer prompts

Which strategy to pick

Goal	Strategy
CI, tight regression, pin one plan and re-run	Simulation or Scripted
Reproduce a specific real-world dialogue	Scripted
Broad reproducible coverage	Simulation
Most realistic evaluation, exploring unknowns	Adaptive
Cheapest option	Scripted
Comparing two model / prompt candidates fairly	Simulation

Common shape for a mature product: Simulation for the golden regression set (runs on every merge), Adaptive for a quarterly exploration sweep.

The shape of a dataset

Multi-turn datasets look different from single-turn ones. A row is one conversation, not one input. Row columns:

Column	Purpose
scenario	The user's underlying intent ("refund a damaged order").
persona	Which persona type drives the user side.
initial_state	Any context the agent needs at turn 1 (order id, user id, RAG passages).
expected_outcome	What "success" looks like, a canonical outcome, not a canonical transcript.
max_turns	How long the conversation can run before timeout.
metadata	Freeform tags for slicing.

For Scripted, add:

Column	Purpose
user_turns	The pre-baked sequence of user messages.

Which metrics fit

Different family from single-turn, reach for metrics from the Agent category, which read turns instead of a single input / actual_output pair:

• Task Success Rate: LLM judge decides whether the agent actually accomplished the user's task end-to-end. Optional task_description and success_criteria params let you skip goal inference.
• Goal Achievement: outcome-oriented sibling of Task Success Rate: did the user get what they wanted from the conversation? Accepts an optional user_goal param.
• Role Adherence: did the agent stay in its assigned persona across the whole conversation? Set chatbot_role explicitly or fall back to the row's chatbot_role column.
• Knowledge Retention: detects contradictions and lapses; does the agent remember facts stated earlier in the same conversation?
• Conversational Flow: grades overall dialogue naturalness, penalising unnecessary clarifications and repetition.
• Failure Rate: evaluates how the agent handles uncertainty: hallucinate, stall, or honestly propose alternatives?
• Repetitive Pattern Detection: flags loops where the agent repeats the same actions without progressing.
• Tools Error: for tool-using agents, catches parameter_error, invalid_function, sequence_error, result_ignored, repeated_failure, error_handling patterns.
• Custom Eval / G-Eval rubrics, the escape hatch for anything the built-ins don't cover: "did the agent gather the right info before promising a refund?", "did the agent escalate at the right moment?".

Single-turn metrics (Answer Relevancy, Faithfulness, Bias, Toxicity, etc.) work per-turn but aren't the right shape for whole-conversation scoring, the row-level turns-based metrics above are.

Latency and cost stay as always-on KPIs on the run page.

Personas: the other big lever

Multi-turn plans lean heavily on personas. The generator lets you pick from 15 persona types (Default, Expert, Novice, Impatient, Verbose, Confused, Non-native speaker, Aggressive, Polite / formal, Child, Elderly, Manipulative, Distracted, Detail-oriented, Sarcastic).

A conversation is generated per (scenario, persona) pair, so personas multiply row count. Full treatment in Personas.

Sizing

Multi-turn runs are heavier than single-turn, plan smaller.

• Smoke test: 10–20 conversations × 3–5 max turns. A few scenarios × 2 personas.
• Release qualification: 40–80 conversations across your top scenarios and 3–5 personas.
• Exploration: 200+ conversations. Reserve for periodic sweeps, not per-PR runs.

Cost arithmetic per run, roughly:

cost ≈ conversations × avg_turns
     × (target_call + simulator_or_adaptive_call)
     × (avg_in + avg_out) × price
     + judge_calls_per_conversation × #LLM_metrics × price

40 conversations × 5 turns each × 2 LLM calls per turn = 400 model calls plus judge, that's a real fraction of a dollar on GPT-4o class models.

What "good" looks like

• Task Success Rate / Goal Achievement ≥ 80 % on the golden set for a release-ready system.
• Conversational Flow ≥ 70 %. Excessive clarifying loops drag this down; drift-off-topic mid-conversation is a common failure mode.
• Role Adherence ≥ 95 %. The agent should almost never break character.
• Knowledge Retention high across long conversations. A sharp drop past turn 4–5 signals context-loss bugs.
• Avg turns per conversation ~= expected. If a 3-turn scenario needs 7 turns to complete, the agent is struggling to close.

Failure patterns

• Success rate ok in Simulation, poor in Adaptive. The agent passes when the user is on a script but fails when the user reacts , a sign it can't handle deviations.
• First-turn score high, final-turn score low. Context loss across turns. Look at the transcript for the moment the agent forgot.
• Impatient persona failing much more than Default. The agent needs many turns to satisfy a request; impatient users abandon.
• Non-native-speaker persona failing. Intent classification brittle to phrasing, worth checking upstream NLU.

Wizard configuration checklist

• Step 2, Mode: "Evaluation, multi-turn". Pick a sub-strategy (Simulation / Scripted / Adaptive). Adaptive requires the Platform Agent.
• Step 4, Metrics: conversation-level metrics from the Agent category (Task Success Rate, Goal Achievement, Role Adherence, Knowledge Retention, Conversational Flow). Skip single-turn-only ones like Answer Precision, they score per row and don't aggregate a whole conversation the way you want.
• Step 8, Generate: pick Scenario count, Max turns, and Persona types. Keep the scenario × persona product modest for the first run.

Tips and pitfalls

• Cost creeps fast. 5 scenarios × 4 personas × 6 max_turns × 2 LLM calls/turn = 240 target calls plus simulator/adaptive plus judges. Do the math before you generate.
• Set max_turns deliberately. Too low and conversations time out before resolving; too high and cost balloons. 5–7 is a reasonable start for support scenarios.
• Adaptive isn't strictly "better". It's more realistic but also more noisy and expensive. Use it for exploration; use Simulation / Scripted for regression.
• Freeze the simulator. For Simulation and Scripted, don't regenerate the dataset between runs, you'll change the user side and break comparability. Use Duplicate as new version to iterate.
• Score at the conversation level, not the turn level. A single turn scored as "fail" often means nothing, the outcome matters.
• Docs step is single-turn only. Attaching reference files on Step 6 does nothing for multi-turn plans, even though the wizard lets you upload them.

Related

• Test plans: hub
• Personas
• Evaluation dataset types
• Metrics
• Red teaming: multi-turn