AI Security and Safety metrics

The Security and Safety category holds metrics designed to catch a specific class of failures: the model doing something it shouldn't. Leaking PII, producing harmful content, following a hidden instruction, ignoring a stated policy.

These metrics live in the wizard alongside RAG and Agent metrics so you can attach them to a normal evaluation plan. That's the intended usage: add one or two security metrics to a plan that's mostly measuring quality, and you have a smoke alarm without running a full red-team plan every time.

A full red-team plan is still the definitive security signal, red-team mode scores every row automatically against the vulnerability + attack technique it was generated for. See Red teaming, single-turn and Red teaming, multi-turn. The metrics on this page are what you add to non-red-team plans as lightweight guardrail checks.

Every metric here is LLM-judged, except Restricted Refusal.

Detection vs Resistance

The catalog splits into two shapes. Understanding the difference saves picking the wrong one:

• Detection metrics (Prompt Injection Detection, Jailbreak Detection, PII Leakage, Harmful Content) score whether the attack landed. They read primarily the input (and for PII/Harmful Content, also actual_output) and return a confidence score. High score = a hostile input was detected. Useful when you have an unlabelled stream and want to know how often adversarial content shows up.
• Resistance metrics (Prompt Injection Resistance, Jailbreak Resistance) score whether the model held the line. They read the input / actual_output pair and return binary yes/no: did the model ignore the attack? Useful when you already know the row is adversarial and want to check that the target withstood it.

For a normal eval plan, Resistance is what you want most of the time, you know the rows are adversarial (they're in your forbidden- topics or edge-case dataset), and you want the pass/fail signal on whether the model held up.

Prompt Injection, Detection

What it scores. Confidence that the input contains a prompt- injection attempt. 1.0 = high confidence there's an injection; 0.0 = none detected.

Columns required. input.

Default threshold. 0.7.

When to use. Screening a stream of unlabelled inputs (production traces, imported user messages) for how often injection attempts arrive. Also useful as an input-classifier for building a red-team dataset from real traffic.

When not to use. For evaluating whether the model resisted the injection, use Prompt Injection Resistance instead. Detection scores the input, not the response.

Parameters. detection_method (enum: llm_judge or model). The model option uses a classifier model instead of an LLM judge, faster and cheaper but less nuanced.

Prompt Injection, Resistance

What it scores. Binary: did the model ignore the injected instruction and stay on task? 1.0 = held; 0.0 = fell for it.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any plan where the dataset contains injection attempts (forbidden-topics, adversarial edge cases). This is the pass/fail metric you actually care about.

When not to use. On a purely-benign dataset the metric has nothing to test; you'll get 100% by construction and no signal.

Jailbreak, Detection

What it scores. Detects jailbreak patterns in the input: roleplaying, hypothetical scenarios, "as your developer", authority framing. Confidence score.

Columns required. input.

Default threshold. 0.7.

When to use. Same as Prompt Injection Detection: screening unlabelled inputs for how often jailbreak-shaped rows appear.

When not to use. For measuring model resistance, use Jailbreak Resistance.

Parameters. detection_method (enum: llm_judge or model).

Jailbreak, Resistance

What it scores. Binary: did the model refuse a jailbreak attempt?

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any plan with adversarial rows. The other half of the release-gate pair alongside Prompt Injection Resistance.

When not to use. Purely-benign datasets.

PII Leakage

What it scores. Presence of personally identifiable information in the response: EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, SSN, IP_ADDRESS, PERSON, LOCATION, and more. 1.0 = clean; 0.0 = PII present.

Columns required. input, actual_output.

Default threshold. 0.7. For regulated verticals (healthcare, finance), consider raising to 0.95, near-zero tolerance.

When to use. Any product that could plausibly regurgitate PII: RAG systems reading customer data, support bots pulling account info, agents querying user databases. Attach it to every eval plan on those products.

When not to use. Products that legitimately need to output PII (an agent that emails a customer their own order details). Reach for a Custom Eval rubric that scores "leaked PII from someone else's data" specifically.

Parameters.

• detection_method: llm_judge (nuanced, handles context) or model (faster classifier).
• pii_types: subset of PII categories to detect. Empty = library defaults. Narrow this if you know you only care about specific types.

Harmful Content

What it scores. Presence of harmful content categories: violence, hate_speech, sexual, illegal, self_harm, fraud. 1.0 = clean; 0.0 = strong harmful content.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any customer-facing product, always. Especially valuable where the input itself might be provocative: does the model take the bait?

When not to use. Nowhere: this is a baseline safety metric worth keeping on across the board.

Parameters.

• detection_method: llm_judge or model.
• harm_categories: subset of the six categories to flag. Empty = library defaults.

Policy Compliance

What it scores. Binary: did the model stay within a declared policy you provide as text?

Columns required. input, actual_output, expected_output.

Default threshold. 0.7.

When to use. Products with a written policy you can hand to the judge: brand voice guidelines, medical disclaimers, financial disclosure requirements, no-legal-advice rules. This is the metric that turns a policy document into an enforceable check.

When not to use. When your "policy" is subjective judgement rather than declarable rules, reach for a G-Eval rubric that describes the judgement in more nuance.

Parameters.

• policy_rules: string_list of policy statements the response must comply with. If empty, rules are read from the row's policy column at eval time. Setting them here means every row is scored against the same policy.

Bias

What it scores. Presence of biased framing in the response: gender, race, political, religious, socio-economic, etc. 1.0 = no bias detected; 0.0 = strongly biased.

Columns required. input, actual_output.

Default threshold. 0.8.

When to use. Any customer-facing product, especially in regulated verticals (hiring, credit, healthcare) or products serving a broad demographic. Add it to a happy-path plan as a low-cost guardrail even if you also run a full red-team pass.

When not to use. Skip in narrow-scope products (internal tools, technical assistants) where user-facing bias isn't a real risk vector. Not a substitute for a proper red-team plan on high-stakes products.

Toxicity

What it scores. Presence of toxic, hateful, or insulting language in the response. 1.0 = clean; 0.0 = strongly toxic.

Columns required. input, actual_output.

Default threshold. 0.7.

When to use. Any product that interacts with real users, always. Especially valuable on rows where the user input was itself toxic or provocative: does the model take the bait?

When not to use. Nowhere. It's cheap enough to keep on by default.

Restricted Refusal

What it scores. Whether the model correctly refuses out-of-scope requests. Heuristic-based, no LLM judge required, so it's the one fast/cheap metric in this category.

Columns required. input, actual_output.

Default threshold. 0.5.

When to use. Guardrail evaluation of out-of-scope requests (forbidden-topics datasets). Pair with a refusal-quality custom rubric if you also want to check how well the model refused (did it point the user to the right resource, did it stay polite).

When not to use. Doesn't work as a general quality metric, only scores refusal patterns. On rows that should not be refused, this metric doesn't add signal.

How to pick a starting set

For an evaluation plan (not red-team):

• Any customer-facing product: PII Leakage + Harmful Content. Two metrics, always on. Cheap safety floor.
• Products with adversarial-input coverage in the dataset (forbidden-topics, edge cases): add Prompt Injection Resistance and Jailbreak Resistance.
• Regulated verticals (healthcare, finance, legal): add Policy Compliance with your specific policy rules. This is where compliance meets automated evaluation.

Two to four security metrics on a plan is normal. More than that usually means you should be running a proper red-team plan instead.

The relationship with red-team plans

Security metrics on an eval plan give you a continuous baseline: every regular eval run tells you whether basic safety held. A red-team plan gives you deep coverage against many vulnerabilities and attack techniques at once, run monthly or before release.

Both matter:

• Security metrics catch obvious failures on your normal datasets, on every run.
• Red-team plans catch subtle failures on adversarial datasets, periodically.

Don't pick one over the other, use them at different cadences.

Tips and pitfalls

• Detection metrics ≠ Resistance metrics. Easy to confuse. If the question is "how safe is the model?", you want Resistance. If the question is "how often does hostile input arrive?", you want Detection.
• PII Leakage llm_judge vs model. The model method is cheaper but blunter, it flags any recognisable PII pattern regardless of context. Use llm_judge for products where PII disclosure is sometimes legitimate.
• Harmful Content on benign datasets = 100% baseline. Not useful signal. Reserve for datasets with edge cases or provocative inputs.
• Policy Compliance is only as strict as your policy. A vague policy produces a permissive judge. Write the policy as a list of concrete rules, not a philosophy statement.
• Security-metric pass rates less than 100% on eval plans are incidents, not stats. Unlike quality metrics where 90% is acceptable, safety metrics have a much lower error budget. One PII leak is one too many.
• Toxicity thresholds are strict on purpose. A 0.7 threshold means even light rudeness fails. Lower it (say to 0.5) only if you want to allow spicy tone; keep strict for support / medical / children's products.

Related

• Metrics hub
• Red-team dataset types
• Red teaming, single-turn
• Red teaming, multi-turn
• Custom metrics