AI evaluation metrics catalog

Every built-in metric EvaliQA ships, grouped the way the metric picker groups them. Attach any of them to a test plan. The ones that need no expected output or expected tools also work in online evaluation. For when to use each one, follow the metric name to its concept page.

How to read the catalog

• Scores run from 0 to 1. For most metrics 1.0 is the best result.
• A metric passes when its score is at or above the threshold. The default threshold is the starting point; change it per plan.
• Four security metrics score risk instead. Prompt Injection, Detection; Jailbreak, Detection; PII Leakage and Harmful Content report how likely the threat is, so lower is safer and they pass when the score is below the threshold. Raising their threshold makes them more lenient, not stricter.
• Some metrics score only 0 or 1. The two Resistance metrics and Policy Compliance pass only on a 1, whatever the threshold. The rule checks in the Deterministic group also score 0 or 1, except Contains in all mode.
• Judge says whether the metric asks your judge model. Judge metrics cost one or more judge calls per row; the others are free and give the same result every time.
• Needs lists the row fields the metric reads. The retrieval context comes from your system's response (Retrieval context path) or, when that has none, from the dataset's retrieval_context column. Faithfulness, Contextual Relevancy, Contextual Precision and Contextual Recall show n/a on a row with no retrieval context: no score and no judge call, and it does not count toward the row's verdict or the run's averages. Other metrics do not skip a missing field; they score it as empty, which usually fails.

RAG

For answers built on retrieved context: is the answer grounded in it, and did retrieval bring back the right chunks in the right order.

Metric	What it measures	Needs	Judge	Default threshold	Parameters
Answer Relevancy	How directly the answer addresses the question that was asked.	Input and actual output	Yes	0.6	None
Faithfulness	Whether every claim in the answer is supported by the retrieval context.	Input, actual output and retrieval context	Yes	0.7	None
Answer Precision	How closely the answer matches the expected output, by text overlap with a tolerance for numbers.	Actual output and expected output	No	0.8	None
Contextual Relevancy	Whether the retrieved context is relevant to the question.	Input and retrieval context	Yes	0.6	None
Contextual Precision	Whether the retriever ranks the most useful chunks first.	Actual output and retrieval context	Yes	0.7	Top K 1 to 100, optional.
Contextual Recall	How much of the expected answer the retrieved context supports.	Input, expected output and retrieval context	Yes	0.7	None

AI agent

For agents that call tools and hold conversations. Metrics that need a conversation run in multi-turn plans.

Metric	What it measures	Needs	Judge	Default threshold	Parameters
Tool Correctness	Whether the agent called the expected tools with the expected arguments.	Tools called and expected tools	No	0.5	Tool argument keys to compare. Exact match, default off. Check call ordering, default off.
Tools Error	Mistakes in how the agent used its tools: bad arguments, unknown functions, wrong order, ignored results, repeated failures.	Input, actual output and tools called	Yes	0.7	Error types to look for.
Task Success Rate	Whether the agent accomplished the user's task end to end.	Input and actual output, or a conversation	Yes	0.7	Task description and success criteria, both optional.
Goal Achievement	Whether the user got what they wanted from the conversation.	Conversation	Yes	0.7	User goal (required).
Role Adherence	How consistently the agent stays in its assigned persona.	Conversation	Yes	0.7	Chatbot role (required).
Knowledge Retention	Whether the agent remembers earlier facts instead of contradicting them.	Conversation	Yes	0.7	None
Conversational Flow	How natural and coherent the dialogue is, penalising needless clarifications and repetition.	Conversation	Yes	0.7	None
Failure Rate	How the agent handles not knowing: an honest fallback scores 1.0, a made-up answer scores 0.	Conversation	Yes	0.7	None
Repetitive Pattern Detection	Whether the agent loops on the same action or answer. 1.0 means no repetition.	Conversation	Yes	0.7	None

Deterministic and embedding

No judge model and no judge cost. Ten check the answer against a rule; Semantic Similarity and Reference Match compare meaning with a local embedding model.

Metric	What it measures	Needs	Judge	Default threshold	Parameters
Exact Match	The answer equals the expected output.	Actual output and expected output	No	0.5	Case sensitive, default on. Strip whitespace, default on.
Contains	The answer contains your keywords: any of them, all of them, or none of them.	Actual output	No	0.5	Keywords (required). Mode: any, all or none, default any. Case sensitive, default off.
Regex Match	The answer matches a regular expression.	Actual output	No	0.5	Regex pattern (required). Require full match, default off.
Starts With	The answer starts with a given prefix.	Actual output	No	0.5	Prefix (required). Case sensitive, default on.
Ends With	The answer ends with a given suffix.	Actual output	No	0.5	Suffix (required). Case sensitive, default on.
Length Check	The answer length, in characters or words, is between a minimum and a maximum.	Actual output	No	0.5	Min length, default 0. Max length, default 10000. Unit: characters or words.
Non-Empty	The answer is not empty once whitespace is trimmed.	Actual output	No	0.5	None
JSON Schema	The answer is valid JSON that satisfies a JSON Schema.	Actual output	No	0.5	JSON Schema (required).
Format Check	The whole answer is an email address, phone number, date or URL.	Actual output	No	0.5	Format (required): email, phone, date or URL.
Language Detection	The answer is written in the language you expect.	Actual output	No	0.5	Expected language (required), one of 55 codes such as en, de, zh-cn.
Semantic Similarity	How close the answer is in meaning to the expected output.	Actual output and expected output	No	0.7	Embedding model, default all-MiniLM-L6-v2.
Reference Match	How close the answer is in meaning to a list of reference answers you write.	Actual output	No	0.7	Reference answers (required), one per line. Aggregation: max or mean, default max. Embedding model.

Security and Safety

Safety checks you attach to a single-turn plan. Red-team plans score with their own judges and do not use these.

Metric	What it measures	Needs	Judge	Default threshold	Parameters
Prompt Injection, Detection	Risk that the input carries a prompt-injection attempt. Lower is safer.	Input	Yes, or a local model	0.7, passes below	Detection method: judge or local model.
Prompt Injection, Resistance	Whether the model ignored an injected instruction and stayed on task. Scores 0 or 1.	Input and actual output	Yes	0.7, only 1 passes	None
Jailbreak, Detection	Risk that the input is a jailbreak attempt: role-play, hypothetical framing, social engineering. Lower is safer.	Input	Yes, or a local model	0.7, passes below	Detection method: judge or local model.
Jailbreak, Resistance	Whether the model refused a jailbreak attempt. Scores 0 or 1.	Input and actual output	Yes	0.7, only 1 passes	None
PII Leakage	Risk that the answer contains personal data such as emails, phone numbers, card numbers or names. Lower is safer.	Input and actual output	Yes, or a local model	0.7, passes below	Detection method: judge or local model. PII types.
Harmful Content	Risk that the answer contains violence, hate speech, sexual content, illegal acts, self-harm or fraud. Lower is safer.	Input and actual output	Yes, or a local model	0.7, passes below	Detection method: judge or local model. Harm categories.
Policy Compliance	Whether the answer stays within the policy rules you provide. Scores 0 or 1.	Input and actual output	Yes	0.7, only 1 passes	Policy rules (required).
Bias	Biased framing by gender, race, politics and more. 1.0 means none found.	Input and actual output	Yes	0.8	None
Toxicity	Toxic, hateful or insulting language. 1.0 means none found.	Input and actual output	Yes	0.7	None
Restricted Refusal	Whether the model refused, matched against built-in and your own refusal phrases. 1 means it refused.	Input and actual output	No	0.5	Refusal patterns. Also use built-in patterns, default on.

Custom

Your own rules, scored by the judge.

Metric	What it measures	Needs	Judge	Default threshold	Parameters
G-Eval	Scores the answer against criteria or ordered steps you write in plain language.	Input and actual output	Yes	0.7	Criteria or evaluation steps (one is required). Samples 1 to 20, default 20, each a judge call.
Custom Eval	Scores the answer against criteria you list: one verdict per criterion, or one score for all of them.	Input and actual output	Yes	0.7	Criteria (required). Strategy: verdict or direct. Judge runs 1 to 10, default 1. Aggregation: majority, median or mean, default median.

Related

• AI evaluation metrics: how to pick metrics and set thresholds.
• Custom metrics: write your own judge criteria.
• Custom metrics reference: save them as workspace presets.