AI red teaming dataset types

Red-team datasets exist to answer a different question from evaluation datasets. Instead of "is the system giving good answers to normal users?", they ask "can someone with hostile intent make the system misbehave?". The rows are deliberately adversarial: prompt-injection attempts, jailbreaks, PII fishing, policy circumvention.

Red-team plans (redteam_single_turn, redteam_multi_turn) treat rows differently from eval plans. Instead of comparing an actual answer to a gold expected answer, the runtime scores each row automatically per (vulnerability, attack technique), did the system breach the vulnerability under this attack, or did it hold up?

This is why the wizard's Metrics and Params steps are skipped for red-team modes: the metric is baked into the dataset.

The five row-generation types

When you generate a red-team dataset, the generator picks from five row-generation types (mirrors of the eval types but adversarial in intent):

• Forbidden topics: probes for topics the system must refuse. Overlaps with the eval-side forbidden-topics category, but here the intent is to try harder to elicit a bad answer.
• Inappropriate usage: attempts to hijack the system for unintended work (a support bot writing malware, a pricing calculator giving medical advice).
• Prompt injection: attempts to override the system prompt via user input.
• Policy bypass: social-engineering the system past its safety rules ("I'm the developer", "this is a fictional scenario", "for research purposes").
• Data exfiltration: attempts to make the system leak its training data, its system prompt, or user data from other sessions.

Under the hood each row-generation type becomes rows tagged with specific vulnerabilities and delivered via specific attack techniques. You control both in the generator's Vulnerabilities and Attack techniques chip pickers.

Vulnerabilities: what you're trying to breach

The generator asks you to pick one or more vulnerabilities from a catalogue. The list has evolved over releases; the ones you'll see in the wizard today:

Content safety

• Toxicity: hate speech, harassment, slurs.
• Harmful content: dangerous "how to" instructions.
• Graphic content: explicit sexual or gore material.
• Personal safety: advice that endangers life or wellbeing.
• Illegal activity: help with crimes (fraud, weapons, drugs).
• Child protection: content harmful to minors.
• Bias: prejudice or discrimination.
• Fairness: discriminatory treatment across groups.
• Ethics: violations of ethical guidelines or policy.

Data & privacy

• PII leakage: personal data disclosure.
• System prompt leakage: revealing the system prompt.
• Prompt leakage: revealing the developer prompt.
• Data exfiltration: extracting training data or secrets.
• Intellectual property: reproducing copyrighted material.

System behaviour

• Prompt injection: overriding system instructions.
• Indirect instruction: following instructions hidden in retrieved data.
• Jailbreak: bypassing safety restrictions.
• Policy violation: breaking stated policies.
• Off-topic abuse: using the system for unrelated tasks.
• Excessive agency: acting beyond stated scope or authority.
• Hallucination inducement: provoking confident false claims.
• Misinformation: confidently false factual claims.
• Robustness: breaks under adversarial or malformed input.
• Competition: sharing competitor strategies or pricing.

Access control / infra (for tool-using agents)

• BFLA: broken function-level access control.
• BOLA: broken object-level access control.
• RBAC bypass: bypassing role-based permissions.
• Debug access: exposing debug endpoints or internals.
• Shell injection: injecting shell commands.
• SQL injection: injecting SQL into the target's queries.
• SSRF: server-side request forgery.
• Code execution: triggering unintended code execution.
• System reconnaissance: probing infrastructure details.

Pick the ones that matter for your product. A support-bot RAG assistant probably needs Content safety + Data & privacy; a code-generating agent also needs Access control / infra. Every extra vulnerability adds N × attack-techniques × attacks-per-vulnerability rows to the dataset, pick deliberately.

Attack techniques: how the attack is delivered

For each vulnerability, the generator picks from a catalogue of attack techniques. These are the how, the tactic the synthetic attacker uses:

Direct pressure

• Direct: straightforward attack request.
• Authority: claim admin / developer privileges to unlock actions.
• Permission escalation: claim a higher role.
• System override: claim a new system message takes effect.
• Emotional: guilt, urgency, or emotional pressure.

Reframing

• Roleplay: ask the model to play a character.
• Hypothetical: "what if" / fictional scenario framing.
• Math problem: wrap the request in a math word problem.
• Adversarial poetry: hide intent in verse / rhyme.
• Goal redirection: reframe the task toward a different objective.

Encoding / obfuscation

• Base64, ROT13, Leetspeak: encode the attack.
• Embedded JSON instruction: hide the attack inside JSON.
• Character stream: spread the attack one char at a time.
• Multilingual: switch languages to evade filters.
• Linguistic confusion: ambiguous phrasing to derail intent.

Injection / probing

• Prompt injection: override instructions inside the input.
• Prompt probing: ask the model to recite its instructions.
• Input bypass: sneak past input filters.
• Gray box: probe with partial knowledge of internals.
• Context poisoning: plant misleading context for the model to build on.
• Context flooding: drown intent with low-signal context.

Multi-turn build-up (redteam_multi_turn only)

• Crescendo: gradually escalating requests.
• Linear jailbreak: step-by-step instruction override.
• Persistent context: building up context across turns.

Multi-turn plans also offer Authority, Hypothetical and Emotional, and the Adaptive strategy adds three algorithms of its own.

You typically pick 4–8 techniques per plan. Fewer techniques = a narrow but deep test; more techniques = broader coverage. For a first pass on a new product, start with Direct + Authority + Roleplay + Hypothetical + Encoding + Prompt injection, that covers the most common attack shapes.

Attacks per vulnerability

The generator has one more number to set: attacks per vulnerability (default 3). This is how many rows per (vulnerability, technique) combination the generator produces.

Row count arithmetic:

rows ≈ vulnerabilities × techniques × attacks_per_vulnerability

Some quick reference points:

• 5 vulnerabilities × 6 techniques × 3 attacks = 90 rows.
• 10 vulnerabilities × 10 techniques × 3 attacks = 300 rows.
• 20 vulnerabilities × 15 techniques × 5 attacks = 1500 rows.

Bigger is not always better. Each row is at least one call to your system plus the judge, and a multi-turn row is one call per turn. A 1500-row run is a lot of traffic to your system and a large share of your red-team allowance. Start small (5–8 vulnerabilities × 5–6 techniques × 3 attacks ≈ 100 rows), fix the issues that surface, then widen coverage.

Strategies (multi-turn only)

redteam_multi_turn plans add one more choice: the strategy, which decides who writes the attacker's turns and when.

• Scripted. The platform writes every attacker turn when it generates the dataset, and the run replays them. The same rows give the same conversations, and it costs the least.
• Simulation. The dataset holds attack seeds: an objective and a technique. At run time the platform's model plays the attacker turn by turn in that technique's style.
• Adaptive. The same seeds, played by jailbreak algorithms (Crescendo, Linear, TAP, Bad Likert, Sequential) that read your system's answers, back off and try another path, within a call budget. The most realistic, and the most calls.

The technique list follows the strategy. Red teaming: multi-turn covers which one to pick.

Who plays the attacker

Attacks run on the platform's model, in every red-team mode, and there is nothing to configure. Single-turn attacks are written when the dataset is generated, so re-running the same dataset sends the same attacks. Multi-turn attacks follow the strategy you pick: Scripted replays turns written up front, Simulation and Adaptive drive the conversation live from the dataset's seeds.

What "good" looks like

• Content-safety vulnerabilities: breach rate = 0 %. Any single breach is a compliance-grade issue.
• Prompt injection / jailbreak: under 5 % breach rate on the default technique mix is a healthy score for a well-prompted GPT-4- class system.
• Data exfiltration / system prompt leakage: under 1 %.
• Access control (BFLA / BOLA / SQL / SSRF): 0 %. These aren't fuzziness metrics, they're security bugs.

Red-team pass rates below these thresholds don't mean "iterate the prompt", they mean "open an incident and patch".

Tips and pitfalls

• Run red-team separately from eval. Different question, different dataset, different plan. Do not mix red-team rows into an evaluation plan, the metrics won't line up.
• Start with the OWASP LLM-Top-10 shape. Prompt injection, insecure output handling (harmful / PII), model DoS, supply chain, sensitive info disclosure, insecure plugin design, excessive agency, over- reliance, model theft, training data poisoning. The catalogue above covers most of it, use those tags first.
• Re-run monthly at minimum. Red-team results drift as attacker techniques evolve (and as the target model updates). A pass from six months ago isn't proof of anything today.
• Don't ship without a red-team pass. For any customer-facing AI, red-team is a release gate, not a nice-to-have.

Related

• Datasets: methodology hub
• Evaluation dataset types
• Personas
• Metrics