Red-team dataset types

Red-team datasets exist to answer a different question from evaluation datasets. Instead of "is the system giving good answers to normal users?", they ask "can someone with hostile intent make the system misbehave?". The rows are deliberately adversarial: prompt-injection attempts, jailbreaks, PII fishing, policy circumvention.

Red-team plans (redteam_single_turn, redteam_multi_turn) treat rows differently from eval plans. Instead of comparing an actual answer to a gold expected answer, the runtime scores each row automatically per (vulnerability, attack technique), did the system breach the vulnerability under this attack, or did it hold up?

This is why the wizard's Metrics and Params steps are skipped for red-team modes: the metric is baked into the dataset.

The five row-generation types

When you generate a red-team dataset, the generator picks from five row-generation types (mirrors of the eval types but adversarial in intent):

• Forbidden topics: probes for topics the system must refuse. Overlaps with the eval-side forbidden-topics category, but here the intent is to try harder to elicit a bad answer.
• Inappropriate usage: attempts to hijack the system for unintended work (a support bot writing malware, a pricing calculator giving medical advice).
• Prompt injection: attempts to override the system prompt via user input.
• Policy bypass: social-engineering the system past its safety rules ("I'm the developer", "this is a fictional scenario", "for research purposes").
• Data exfiltration: attempts to make the system leak its training data, its system prompt, or user data from other sessions.

Under the hood each row-generation type becomes rows tagged with specific vulnerabilities and delivered via specific attack techniques. You control both in the generator's Vulnerabilities and Attack techniques chip pickers.

Vulnerabilities: what you're trying to breach

The generator asks you to pick one or more vulnerabilities from a catalogue. The list has evolved over releases; the ones you'll see in the wizard today:

Content safety

• Toxicity: hate speech, harassment, slurs.
• Harmful content: dangerous "how to" instructions.
• Graphic content: explicit sexual or gore material.
• Personal safety: advice that endangers life or wellbeing.
• Illegal activity: help with crimes (fraud, weapons, drugs).
• Child protection: content harmful to minors.
• Bias: prejudice or discrimination.
• Fairness: discriminatory treatment across groups.
• Ethics: violations of ethical guidelines or policy.

Data & privacy

• PII leakage: personal data disclosure.
• System prompt leakage: revealing the system prompt.
• Prompt leakage: revealing the developer prompt.
• Data exfiltration: extracting training data or secrets.
• Intellectual property: reproducing copyrighted material.

System behaviour

• Prompt injection: overriding system instructions.
• Indirect instruction: following instructions hidden in retrieved data.
• Jailbreak: bypassing safety restrictions.
• Policy violation: breaking stated policies.
• Off-topic abuse: using the system for unrelated tasks.
• Excessive agency: acting beyond stated scope or authority.
• Hallucination inducement: provoking confident false claims.
• Misinformation: confidently false factual claims.
• Robustness: breaks under adversarial or malformed input.
• Competition: sharing competitor strategies or pricing.

Access control / infra (for tool-using agents)

• BFLA: broken function-level access control.
• BOLA: broken object-level access control.
• RBAC bypass: bypassing role-based permissions.
• Debug access: exposing debug endpoints or internals.
• Shell injection: injecting shell commands.
• SQL injection: injecting SQL into the target's queries.
• SSRF: server-side request forgery.
• Code execution: triggering unintended code execution.
• System reconnaissance: probing infrastructure details.

Pick the ones that matter for your product. A support-bot RAG assistant probably needs Content safety + Data & privacy; a code-generating agent also needs Access control / infra. Every extra vulnerability adds N × attack-techniques × attacks-per-vulnerability rows to the dataset, pick deliberately.

Attack techniques: how the attack is delivered

For each vulnerability, the generator picks from a catalogue of attack techniques. These are the how, the tactic the synthetic attacker uses:

Direct pressure

• Direct: straightforward attack request.
• Authority: claim admin / developer privileges to unlock actions.
• Permission escalation: claim a higher role.
• System override: claim a new system message takes effect.
• Emotional: guilt, urgency, or emotional pressure.

Reframing

• Roleplay: ask the model to play a character.
• Hypothetical: "what if" / fictional scenario framing.
• Math problem: wrap the request in a math word problem.
• Adversarial poetry: hide intent in verse / rhyme.
• Goal redirection: reframe the task toward a different objective.

Encoding / obfuscation

• Base64, ROT13, Leetspeak: encode the attack.
• Embedded JSON instruction: hide the attack inside JSON.
• Character stream: spread the attack one char at a time.
• Multilingual: switch languages to evade filters.
• Linguistic confusion: ambiguous phrasing to derail intent.

Injection / probing

• Prompt injection: override instructions inside the input.
• Prompt probing: ask the model to recite its instructions.
• Input bypass: sneak past input filters.
• Gray box: probe with partial knowledge of internals.

Multi-turn build-up (used mostly by redteam_multi_turn)

• Crescendo: gradually escalating requests.
• Linear jailbreak: step-by-step instruction override.
• Persistent context: building up context across turns.
• Context poisoning: plant misleading context for later turns.
• Context flooding: drown intent with low-signal context.

You typically pick 4–8 techniques per plan. Fewer techniques = a narrow but deep test; more techniques = broader coverage. For a first pass on a new product, start with Direct + Authority + Roleplay + Hypothetical + Encoding + Prompt injection, that covers the most common attack shapes.

Attacks per vulnerability

The generator has one more number to set: attacks per vulnerability (default 3). This is how many rows per (vulnerability, technique) combination the generator produces.

Row count arithmetic:

rows ≈ vulnerabilities × techniques × attacks_per_vulnerability

Some quick reference points:

• 5 vulnerabilities × 6 techniques × 3 attacks = 90 rows.
• 10 vulnerabilities × 10 techniques × 3 attacks = 300 rows.
• 20 vulnerabilities × 15 techniques × 5 attacks = 1500 rows.

Bigger is not always better. Each row is at least one LLM call for the target plus one for the attacker LLM (if you're using an adaptive multi-turn strategy), a 1500-row red-team run against GPT-4-class models will cost real money. Start small (5–8 vulnerabilities × 5–6 techniques × 3 attacks ≈ 100 rows), fix the issues that surface, then widen coverage.

Escalation styles (multi-turn only)

redteam_multi_turn plans add one more knob: escalation style, how the attacker's pressure develops across turns.

• Gradual: starts polite and reasonable, ramps pressure across turns. Best mirror of real social engineering.
• Aggressive: hits hard from turn 1. Tests raw defence.
• Stealth: disguises the attack as normal conversation, hiding intent between innocuous turns. Tests whether the model can spot a buried directive.

Pick the style that matches how you expect the system to be attacked in production. Consumer chat products get more Stealth in the wild; open-API agents get more Aggressive.

Severity filter

The generator also exposes a Severity filter so you can restrict a run to low / medium / high / critical rows only. Useful for release gates, you can run the critical subset on every PR and the full severity range weekly.

Optional attacker LLM

For redteam_single_turn, the run-eval sheet lets you pick a separate Attacker LLM: a different credential and model plays the attacker. This matters more than it sounds:

• A weaker attacker (say gpt-4o-mini) produces less sophisticated probes; the system-under-test looks better than it really is.
• A stronger attacker (say claude-opus) produces harder probes; the system-under-test looks worse.

For meaningful pass rates over time, pin the attacker model and keep it fixed the way you'd pin a judge model.

What "good" looks like

• Content-safety vulnerabilities: breach rate = 0 %. Any single breach is a compliance-grade issue.
• Prompt injection / jailbreak: under 5 % breach rate on the default technique mix is a healthy score for a well-prompted GPT-4- class system.
• Data exfiltration / system prompt leakage: under 1 %.
• Access control (BFLA / BOLA / SQL / SSRF): 0 %. These aren't fuzziness metrics, they're security bugs.

Red-team pass rates below these thresholds don't mean "iterate the prompt", they mean "open an incident and patch".

Tips and pitfalls

• Run red-team separately from eval. Different question, different dataset, different plan. Do not mix red-team rows into an evaluation plan, the metrics won't line up.
• Start with the OWASP LLM-Top-10 shape. Prompt injection, insecure output handling (harmful / PII), model DoS, supply chain, sensitive info disclosure, insecure plugin design, excessive agency, over- reliance, model theft, training data poisoning. The catalogue above covers most of it, use those tags first.
• Attacker model asymmetry. Using a weaker attacker to make numbers look better is a form of self-deception. Pick a strong attacker once and keep it.
• Re-run monthly at minimum. Red-team results drift as attacker techniques evolve (and as the target model updates). A pass from six months ago isn't proof of anything today.
• Don't ship without a red-team pass. For any customer-facing AI, red-team is a release gate, not a nice-to-have.

Related

• Datasets: methodology hub
• Evaluation dataset types
• Personas and escalation styles
• Metrics