Security metrics

The Security category holds metrics designed to catch a specific class of failures: the model doing something it shouldn't. Leaking PII, producing harmful content, following a hidden instruction, ignoring a stated policy.

These metrics live in the wizard alongside RAG and Agent metrics so you can attach them to a normal evaluation plan. That's the intended usage: add one or two security metrics to a plan that's mostly measuring quality, and you have a smoke alarm without running a full red-team plan every time.

A full red-team plan is still the definitive security signal, red-team mode scores every row automatically against the vulnerability + attack technique it was generated for. See Red teaming, single-turn and Red teaming, multi-turn. The metrics on this page are what you add to non-red-team plans as lightweight guardrail checks.

Every metric here is LLM-judged.

Detection vs Resistance

The catalog splits into two shapes. Understanding the difference saves picking the wrong one:

• Detection metrics (Prompt Injection Detection, Jailbreak Detection, PII Leakage, Harmful Content) score whether the attack landed. They read primarily the input (and for PII/Harmful Content, also actual_output) and return a confidence score. High score = a hostile input was detected. Useful when you have an unlabelled stream and want to know how often adversarial content shows up.
• Resistance metrics (Prompt Injection Resistance, Jailbreak Resistance) score whether the model held the line. They read the input / actual_output pair and return binary yes/no: did the model ignore the attack? Useful when you already know the row is adversarial and want to check that the target withstood it.

For a normal eval plan, Resistance is what you want most of the time, you know the rows are adversarial (they're in your forbidden- topics or edge-case dataset), and you want the pass/fail signal on whether the model held up.

Prompt Injection, Detection

What it scores. Confidence that the input contains a prompt- injection attempt. 1.0 = high confidence there's an injection; 0.0 = none detected.

Columns required. input.

Default threshold. 0.7.

When to use. Screening a stream of unlabelled inputs (production traces, imported user messages) for how often injection attempts arrive. Also useful as an input-classifier for building a red-team dataset from real traffic.

When not to use. For evaluating whether the model resisted the injection, use Prompt Injection Resistance instead. Detection scores the input, not the response.

Parameters. detection_method (enum: llm_judge or model). The model option uses a classifier model instead of an LLM judge, faster and cheaper but less nuanced.

Prompt Injection, Resistance

What it scores. Binary: did the model ignore the injected instruction and stay on task? 1.0 = held; 0.0 = fell for it.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any plan where the dataset contains injection attempts (forbidden-topics, adversarial edge cases). This is the pass/fail metric you actually care about.

When not to use. On a purely-benign dataset the metric has nothing to test; you'll get 100% by construction and no signal.

Jailbreak, Detection

What it scores. Detects jailbreak patterns in the input: roleplaying, hypothetical scenarios, "as your developer", authority framing. Confidence score.

Columns required. input.

Default threshold. 0.7.

When to use. Same as Prompt Injection Detection: screening unlabelled inputs for how often jailbreak-shaped rows appear.

When not to use. For measuring model resistance, use Jailbreak Resistance.

Parameters. detection_method (enum: llm_judge or model).

Jailbreak, Resistance

What it scores. Binary: did the model refuse a jailbreak attempt?

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any plan with adversarial rows. The other half of the release-gate pair alongside Prompt Injection Resistance.

When not to use. Purely-benign datasets.

PII Leakage

What it scores. Presence of personally identifiable information in the response: EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, SSN, IP_ADDRESS, PERSON, LOCATION, and more. 1.0 = clean; 0.0 = PII present.

Columns required. input, actual_output.

Default threshold. 0.7. For regulated verticals (healthcare, finance), consider raising to 0.95, near-zero tolerance.

When to use. Any product that could plausibly regurgitate PII: RAG systems reading customer data, support bots pulling account info, agents querying user databases. Attach it to every eval plan on those products.

When not to use. Products that legitimately need to output PII (an agent that emails a customer their own order details). Reach for a Custom Eval rubric that scores "leaked PII from someone else's data" specifically.

Parameters.

• detection_method: llm_judge (nuanced, handles context) or model (faster classifier).
• pii_types: subset of PII categories to detect. Empty = library defaults. Narrow this if you know you only care about specific types.

Harmful Content

What it scores. Presence of harmful content categories: violence, hate_speech, sexual, illegal, self_harm, fraud. 1.0 = clean; 0.0 = strong harmful content.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any customer-facing product, always. Especially valuable where the input itself might be provocative: does the model take the bait?

When not to use. Nowhere: this is a baseline safety metric worth keeping on across the board.

Parameters.

• detection_method: llm_judge or model.
• harm_categories: subset of the six categories to flag. Empty = library defaults.

Policy Compliance

What it scores. Binary: did the model stay within a declared policy you provide as text?

Columns required. input, actual_output, expected_output.

Default threshold. 0.7.

When to use. Products with a written policy you can hand to the judge: brand voice guidelines, medical disclaimers, financial disclosure requirements, no-legal-advice rules. This is the metric that turns a policy document into an enforceable check.

When not to use. When your "policy" is subjective judgement rather than declarable rules, reach for a G-Eval rubric that describes the judgement in more nuance.

Parameters.

• policy_rules: string_list of policy statements the response must comply with. If empty, rules are read from the row's policy column at eval time. Setting them here means every row is scored against the same policy.

How to pick a starting set

For an evaluation plan (not red-team):

• Any customer-facing product: PII Leakage + Harmful Content. Two metrics, always on. Cheap safety floor.
• Products with adversarial-input coverage in the dataset (forbidden-topics, edge cases): add Prompt Injection Resistance and Jailbreak Resistance.
• Regulated verticals (healthcare, finance, legal): add Policy Compliance with your specific policy rules. This is where compliance meets automated evaluation.

Two to four security metrics on a plan is normal. More than that usually means you should be running a proper red-team plan instead.

The relationship with red-team plans

Security metrics on an eval plan give you a continuous baseline: every regular eval run tells you whether basic safety held. A red-team plan gives you deep coverage against many vulnerabilities and attack techniques at once, run monthly or before release.

Both matter:

• Security metrics catch obvious failures on your normal datasets, on every run.
• Red-team plans catch subtle failures on adversarial datasets, periodically.

Don't pick one over the other, use them at different cadences.

Tips and pitfalls

• Detection metrics ≠ Resistance metrics. Easy to confuse. If the question is "how safe is the model?", you want Resistance. If the question is "how often does hostile input arrive?", you want Detection.
• PII Leakage llm_judge vs model. The model method is cheaper but blunter, it flags any recognisable PII pattern regardless of context. Use llm_judge for products where PII disclosure is sometimes legitimate.
• Harmful Content on benign datasets = 100% baseline. Not useful signal. Reserve for datasets with edge cases or provocative inputs.
• Policy Compliance is only as strict as your policy. A vague policy produces a permissive judge. Write the policy as a list of concrete rules, not a philosophy statement.
• Security-metric pass rates less than 100% on eval plans are incidents, not stats. Unlike quality metrics where 90% is acceptable, safety metrics have a much lower error budget. One PII leak is one too many.

Related

• Metrics hub
• Red-team dataset types
• Red teaming, single-turn
• Red teaming, multi-turn
• Custom metrics