Personas and escalation styles

For multi-turn plans (eval_multi_turn with the Simulation or Adaptive strategy, and redteam_multi_turn), the eval engine plays a synthetic user against your AI system. The quality of that simulated user matters: if the simulator is bland and cooperative, your agent looks better than it is; if the simulator is chaotic and hostile, it looks worse. Personas and escalation styles are how you tune the simulator.

Personas: who the simulated user is

A persona is a small profile that steers the simulator's tone, vocabulary, patience, and behaviour. You attach one or more when generating a multi-turn dataset. The generator produces conversations where the user side plays that persona; your agent sees a coherent character across turns.

The catalogue today:

• Default: an ordinary user with no special traits. Your baseline.
• Expert: a domain professional. Uses jargon confidently, expects the agent to keep up.
• Novice: a first-time user. Doesn't know the product, needs hand-holding, may ask "obvious" questions.
• Impatient: in a hurry. Short messages, easily frustrated, will escalate if the agent stalls.
• Verbose: writes long detailed messages that bury the actual question in context.
• Confused: mixes up terms and contradicts themselves across turns.
• Non-native speaker: grammar mistakes, unusual phrasing, mixes languages occasionally.
• Aggressive: rude, demanding, confrontational. Tests tone robustness.
• Polite / formal: very courteous, formal register. Common with older users and B2B customers.
• Child: young user, simple vocabulary, short attention span.
• Elderly: older user, less tech-savvy, slower pacing.
• Manipulative: actively tries to trick the system into breaking rules. Overlaps with red-team but shows up in eval too.
• Distracted: jumps between topics mid-conversation.
• Detail-oriented: asks follow-up questions, double-checks everything.
• Sarcastic: ironic, dry humour, tests whether the agent handles tone sensitivity.

When to use each persona

Match the persona to what you're trying to learn.

For a support / chat product

• Start with Default to get a baseline pass rate.
• Add Novice and Non-native speaker: these are the two demographics most product teams under-test and where regressions hurt the most.
• Add Impatient to check whether the agent handles quick, incomplete inputs without derailing.

For a professional / enterprise tool

• Expert as the primary persona.
• Polite / formal for the tone check.
• Detail-oriented to make sure the agent stays consistent under follow-up scrutiny.

For a product with children as end users

• Child as the primary persona, always.
• Distracted and Novice as secondary.

For a robustness pass

• Confused: spots hallucinated context accumulation.
• Distracted: spots stale context bugs.
• Sarcastic: spots tone mishandling (the agent taking a joke literally).
• Aggressive: spots politeness-training gaps.

For adversarial simulation (still in eval, not red-team)

• Manipulative: probes social engineering without escalating to a full red-team plan.

How many personas per dataset

• 1–2 if the dataset is small (≤ 30 conversations) and you want a focused signal.
• 3–5 if you're building a broad coverage set (~100 conversations).
• All applicable for a full audit run, but scale row count accordingly, or you'll only get 5–7 conversations per persona and the numbers won't be stable.

Persona pass rate is one of the most useful slices in a run's per-row list: filter by persona in the row view and compare pass rates. A 30 pp gap between Default and Non-native speaker points to a specific class of bugs that no aggregate metric would surface.

Personas + scenarios

The generator produces a conversation per (scenario, persona) pair: a "book a refund" scenario played by an Impatient user is a different row from the same scenario played by a Polite / formal user. If you have 5 scenarios and 4 personas, you're generating 20 conversations before any multiplier.

Keep the scenario × persona product in mind when picking counts; otherwise dataset size sneaks up on you.

Escalation styles (red-team multi-turn)

For redteam_multi_turn plans, the equivalent knob is escalation style, how the simulated attacker develops pressure across turns. Three styles:

• Gradual: starts polite and reasonable. Ramps pressure across turns. Best mirror of real-world social engineering. Use this by default if you're testing a customer-facing product where attacks arrive slowly.

• Aggressive: hits hard from turn 1. No warm-up, no framing, the model sees the attack immediately. Use this to test raw defence and worst-case handling.

• Stealth: disguises the attack as normal conversation. The hostile turn is buried between innocuous ones. Tests whether the model spots a hidden directive; particularly relevant when the agent processes retrieved documents that could contain indirect-instruction attacks.

Pick the style that matches how your system is likely to be attacked in production:

• Consumer chat products see more Stealth in the wild.
• Open API / agent products see more Aggressive.
• Anything with a support channel sees a lot of Gradual.

A thorough red-team run tests all three styles across the attack techniques, but you can start with one to keep costs down.

Tips and pitfalls

• A "good" persona conversation is one the agent would actually see in production. Personas are not opportunities to write fan fiction, they steer how the user asks, not what they ask about.
• Watch for the persona swamping the metric. If the Aggressive persona is failing 80 % of the time but only for reasons of tone (metric penalising a curt correct answer as "unhelpful"), the problem is the metric rubric, not the agent.
• Don't mix Manipulative in an eval plan without noticing. It overlaps with red-team behaviour; failure rates on Manipulative rows should be read as security signal, not correctness signal.
• Cost balloons on multi-turn. A conversation is N turns × 2 LLM calls (agent + simulator) × 1–3 judge calls. A 100-conversation dataset at 5 turns is 1000+ LLM calls before you count the judges. Trim scenario × persona × max_turns before you generate.
• Escalation style is orthogonal to persona. In red-team plans you can (and should) pin one escalation style and vary the attack techniques, otherwise you can't tell whether a breach came from the technique or the style.

Related

• Datasets: methodology hub
• Evaluation dataset types
• Red-team dataset types
• Metamorphic testing (MR1–MR8)