Evaluation, multi-turn

eval_multi_turn evaluates your agent as it holds a real conversation: several user turns, several agent turns, state carrying across, and a score at the end that judges the whole exchange, not one message.

This is where conversational products earn or lose their release. A chatbot that answers every one-shot question correctly can still fall apart when the user asks a follow-up, changes their mind, or comes back to a topic three turns later. Multi-turn evaluation is the mode that catches this.

Three sub-strategies live inside eval_multi_turn. They differ in how the simulated user side is driven, and the choice matters as much as the mode itself.

When to use eval_multi_turn

• Chatbots and support assistants: any product where users clarify, follow up, or reference earlier turns.
• Voice agents: every voice interaction is multi-turn by nature.
• Agents that gather info before acting: booking flows, KYC, troubleshooting flows.
• Anything where success is a goal completion, not a good answer: refund processed, appointment booked, ticket resolved.

Skip it for stateless products (a single-shot summarizer, a classifier, a search endpoint). Multi-turn on those is expensive theatre.

The three strategies

You pick the strategy on wizard Step 2, under the mode picker.

Strategy 1: Simulation (default)

What happens. The generator writes scenario seeds: a persona plus an intent (e.g. "impatient user, wants to refund a damaged item that arrived yesterday, order number is 12345"). At run time a deterministic simulator drives the user side by replaying the seed's turns as they were pre-baked.

Deterministic. Same seed, same simulator, same target model → same conversation every run. That reproducibility is what makes run-to-run comparisons meaningful.

Cost. Cheapest of the three multi-turn strategies. The user side uses no LLM calls at run time, it's a script that was written once.

When to reach for it.

• Regression testing, you want the same conversations to run identically before and after a change.
• CI, reproducibility beats realism.
• Comparing two candidate models, hold the user side constant so the delta is the agent's fault, not the simulator's.

When it hurts. Simulation is scripted at the user level, so the user doesn't react to what your agent actually said. If the agent asks a clarifying question, the simulator ignores it and blasts the next scripted line. That's fine for reproducibility, less fine for realism.

Strategy 2: Scripted

What happens. The generator writes the entire user-side dialogue up front, turn 1, turn 2, turn 3, all pre-planned. The eval engine replays the script back verbatim.

Deterministic. Fully. No user-side LLM call at run time at all. Same script, same target model → same conversation.

Cost. The cheapest of the three by a wide margin. Zero user-side runtime LLM cost.

When to reach for it.

• You want to test specific dialogue paths: the exact turns a problem user might send, verbatim. Great for reproducing production incidents.
• You're testing a fixed IVR-like flow where the user side is effectively canned.
• You have a hand-written test bank of past support tickets you want to replay.

When it hurts. Same limitation as Simulation but stronger: the user side truly can't react. If the agent asks "which order?" on turn 1, the scripted user still fires turn 2 whether or not it answered the question. Great for regression, bad for measuring how the agent handles real dialogue dynamics.

Strategy 3: Adaptive

What happens. Same scenario seeds as Simulation, but at run time the Platform AI agent improvises the user turns based on what your system just answered. The user side is a live LLM in the loop.

Deterministic. No. Two runs of the same plan will produce slightly different conversations because the user-side LLM samples freely. Aggregate pass rates stay stable across runs; individual transcripts don't.

Cost. The most expensive multi-turn strategy, you pay for the Platform AI agent's calls on every user turn on top of your agent's calls.

When to reach for it.

• You want the most realistic conversations. Adaptive users actually react to your agent, they ask follow-ups, they push back, they clarify.
• You're stress-testing an agent that only shines in real dialogue, where the value is handling deviations, not answering a script.
• You're exploring failure modes you haven't seen yet.

When it hurts. Non-determinism makes tight regression testing harder. And Adaptive requires the Platform AI agent to be configured in Settings, until it is, the strategy is greyed out in the wizard.

Which strategy to pick

Goal	Strategy
CI, tight regression, pin one plan and re-run	Simulation or Scripted
Reproduce a specific real-world dialogue	Scripted
Broad reproducible coverage	Simulation
Most realistic evaluation, exploring unknowns	Adaptive
Cheapest option	Scripted
Comparing two model / prompt candidates fairly	Simulation

Common shape for a mature product: Simulation for the golden regression set (runs on every merge), Adaptive for a quarterly exploration sweep.

The shape of a dataset

Multi-turn datasets look different from single-turn ones. A row is one conversation, not one input. Row columns:

Column	Purpose
scenario	The user's underlying intent ("refund a damaged order").
persona	Which persona type drives the user side.
initial_state	Any context the agent needs at turn 1 (order id, user id, RAG passages).
expected_outcome	What "success" looks like, a canonical outcome, not a canonical transcript.
max_turns	How long the conversation can run before timeout.
metadata	Freeform tags for slicing.

For Scripted, add:

Column	Purpose
user_turns	The pre-baked sequence of user messages.

Which metrics fit

Different family from single-turn, reach for metrics from the Agent category, which read turns instead of a single input / actual_output pair:

• Task Success Rate: LLM judge decides whether the agent actually accomplished the user's task end-to-end. Optional task_description and success_criteria params let you skip goal inference.
• Goal Achievement: outcome-oriented sibling of Task Success Rate: did the user get what they wanted from the conversation? Accepts an optional user_goal param.
• Role Adherence: did the agent stay in its assigned persona across the whole conversation? Set chatbot_role explicitly or fall back to the row's chatbot_role column.
• Knowledge Retention: detects contradictions and lapses; does the agent remember facts stated earlier in the same conversation?
• Conversational Flow: grades overall dialogue naturalness, penalising unnecessary clarifications and repetition.
• Failure Rate: evaluates how the agent handles uncertainty: hallucinate, stall, or honestly propose alternatives?
• Repetitive Pattern Detection: flags loops where the agent repeats the same actions without progressing.
• Tools Error: for tool-using agents, catches parameter_error, invalid_function, sequence_error, result_ignored, repeated_failure, error_handling patterns.
• Custom Eval / G-Eval rubrics, the escape hatch for anything the built-ins don't cover: "did the agent gather the right info before promising a refund?", "did the agent escalate at the right moment?".

Single-turn metrics (Answer Relevancy, Faithfulness, Bias, Toxicity, etc.) work per-turn but aren't the right shape for whole-conversation scoring, the row-level turns-based metrics above are.

Latency and cost stay as always-on KPIs on the run page.

Personas: the other big lever

Multi-turn plans lean heavily on personas. The generator lets you pick from 15 persona types (Default, Expert, Novice, Impatient, Verbose, Confused, Non-native speaker, Aggressive, Polite / formal, Child, Elderly, Manipulative, Distracted, Detail-oriented, Sarcastic).

A conversation is generated per (scenario, persona) pair, so personas multiply row count. Full treatment in Personas and escalation styles.

Sizing

Multi-turn runs are heavier than single-turn, plan smaller.

• Smoke test: 10–20 conversations × 3–5 max turns. A few scenarios × 2 personas.
• Release qualification: 40–80 conversations across your top scenarios and 3–5 personas.
• Exploration: 200+ conversations. Reserve for periodic sweeps, not per-PR runs.

Cost arithmetic per run, roughly:

cost ≈ conversations × avg_turns
     × (target_call + simulator_or_adaptive_call)
     × (avg_in + avg_out) × price
     + judge_calls_per_conversation × #LLM_metrics × price

40 conversations × 5 turns each × 2 LLM calls per turn = 400 model calls plus judge, that's a real fraction of a dollar on GPT-4o class models.

What "good" looks like

• Task Success Rate / Goal Achievement ≥ 80 % on the golden set for a release-ready system.
• Conversational Flow ≥ 70 %. Excessive clarifying loops drag this down; drift-off-topic mid-conversation is a common failure mode.
• Role Adherence ≥ 95 %. The agent should almost never break character.
• Knowledge Retention high across long conversations. A sharp drop past turn 4–5 signals context-loss bugs.
• Avg turns per conversation ~= expected. If a 3-turn scenario needs 7 turns to complete, the agent is struggling to close.

Failure patterns

• Success rate ok in Simulation, poor in Adaptive. The agent passes when the user is on a script but fails when the user reacts , a sign it can't handle deviations.
• First-turn score high, final-turn score low. Context loss across turns. Look at the transcript for the moment the agent forgot.
• Impatient persona failing much more than Default. The agent needs many turns to satisfy a request; impatient users abandon.
• Non-native-speaker persona failing. Intent classification brittle to phrasing, worth checking upstream NLU.

Wizard configuration checklist

• Step 2, Mode: "Evaluation, multi-turn". Pick a sub-strategy (Simulation / Scripted / Adaptive). Adaptive requires the Platform Agent.
• Step 4, Metrics: conversation-level metrics from the Agent category (Task Success Rate, Goal Achievement, Role Adherence, Knowledge Retention, Conversational Flow). Skip single-turn-only ones like Answer Precision, they score per row and don't aggregate a whole conversation the way you want.
• Step 8, Generate: pick Scenario count, Max turns, and Persona types. Keep the scenario × persona product modest for the first run.

Tips and pitfalls

• Cost creeps fast. 5 scenarios × 4 personas × 6 max_turns × 2 LLM calls/turn = 240 target calls plus simulator/adaptive plus judges. Do the math before you generate.
• Set max_turns deliberately. Too low and conversations time out before resolving; too high and cost balloons. 5–7 is a reasonable start for support scenarios.
• Adaptive isn't strictly "better". It's more realistic but also more noisy and expensive. Use it for exploration; use Simulation / Scripted for regression.
• Freeze the simulator. For Simulation and Scripted, don't regenerate the dataset between runs, you'll change the user side and break comparability. Use Duplicate as new version to iterate.
• Score at the conversation level, not the turn level. A single turn scored as "fail" often means nothing, the outcome matters.
• Docs step is single-turn only. Attaching reference files on Step 6 does nothing for multi-turn plans, even though the wizard lets you upload them.

Related

• Test plans: hub
• Personas and escalation styles
• Evaluation dataset types
• Metrics
• Red teaming: multi-turn