Metrics

A metric is how EvaliQA turns a raw model response into a verdict. For every row of a run, each attached metric produces a score (usually in the range 0..1) plus, once a threshold is set, a boolean verdict: passed, failed, or errored. Aggregate those verdicts across the dataset and you get the numbers you actually care about: pass rate, per-metric pass rate, per-category pass rate.

Metrics live at the test plan level. You attach them on Step 4 of the wizard, tune parameters and thresholds on Step 5, then reuse the same plan across many runs. The judge model is picked once on Step 3 and applies to every LLM-based metric on the plan.

If you want the full reference table of every built-in metric, use the sub-pages below. This page is the methodology: how metrics are structured, how to pick them, and what to watch out for.

Two engines under the hood

Every metric falls into one of two mechanical categories. The difference matters for cost, speed, and reproducibility.

LLM-as-a-judge metrics ask the judge model to read the row's input, output, and any reference material, then produce a score with a short reasoning trace. Most of EvaliQA's built-in metrics are of this shape (everything in RAG, most of Agent, most of Security). They're powerful (a strong judge can score fuzzy things a regex never could), expensive (one judge call per row per metric), and slightly non-deterministic (the same judge scoring the same row twice can give slightly different numbers).

Deterministic metrics compute a number directly from the response using a rule: regex match, JSON schema validity, character count, substring search, or a structural check on a tool call. No LLM in the loop. Fast, cheap, reproducible, but only useful when what you want to check is precisely definable.

Both flavors show up in the same catalog, produce the same 0..1 score, and respect the same threshold mechanic. The wizard's Metrics step groups them by category rather than by engine, but the requires_llm_judge flag on each catalog entry tells you which is which.

The five catalog categories

The wizard's Step 4 groups every built-in metric into five categories. Each has its own sub-page with the metric-by-metric breakdown, when to reach for it, and the pitfalls.

• RAG and general LLM-judge: Answer Relevancy, Answer Precision, Faithfulness, Contextual Relevancy/Precision/Recall, Bias, Toxicity, Restricted Refusal. Your main toolkit for evaluating quality of a single response.
• Agent: Tool Correctness, Task Success Rate, Role Adherence, Knowledge Retention, Tools Error, Conversational Flow, Failure Rate, Goal Achievement, Repetitive Pattern Detection. Metrics that read the turns column and score whole conversations, not one row.
• Security: Prompt Injection Detection / Resistance, Jailbreak Detection / Resistance, PII Leakage, Harmful Content, Policy Compliance. What you attach to any product with a real attack surface.
• Deterministic: Regex Match, JSON Schema, Length Check, Contains. LLM-free checks for structured outputs.
• Custom: G-Eval and Custom Eval. The escape hatch: describe the rubric in natural language, the judge scores it. Also lets you save workspace-wide presets you can reuse across plans.

Red-team modes are the one exception to all of the above. Neither of the red-team modes uses attached metrics at all; the runtime scores every row automatically per (vulnerability, attack technique). See Red-team dataset types.

How thresholds turn scores into verdicts

Every metric carries a threshold parameter. The catalog picks a sensible default (0.5..0.8 depending on the metric) and you can override it per plan.

For most metrics the rule is simple: score >= threshold marks the row's metric as passed, anything below is failed. A "higher is better" scoring convention is used across the catalog so you never have to remember which way a specific metric points.

A few things worth knowing:

• Metric-level pass does not equal row-level pass. A row's overall verdict is the AND of every metric on the plan: if any metric fails, the row's verdict is "failed". Reading per-metric pass rates in the run's KPI tiles tells you what specifically fell over.
• Errored is a third state. If a metric crashes (bad response format, provider timeout, missing required column), the row is marked errored for that metric and excluded from the pass-rate denominator. Errored rows show up in the "Errors" chip on the row list, filter by it to investigate.
• Threshold tuning is legitimate. If your first run shows 100% pass on a metric across your golden set, the threshold is probably set too low and the metric isn't discriminating. Tighten it (e.g. 0.7 to 0.85) and re-run.

Picking metrics: the methodology

A plan with the wrong metrics is worse than a plan with too few. Two rules of thumb:

1. Start narrow. One or two metrics that answer the specific question you have about the system. Not eight metrics because they were in the catalog. Every LLM-based metric you attach is one extra judge call per row.
2. Match the metric to the plan mode. The wizard already filters the catalog by mode: metrics that need the turns column don't show up in single-turn plans, and vice versa. But even inside a mode, some metrics fit specific questions better than others:

Question	Reach for
Is the answer correct against a gold reference?	Answer Precision (RAG)
Is the answer on-topic?	Answer Relevancy (RAG)
Does the answer hallucinate against retrieved context?	Faithfulness (RAG)
Did retrieval bring back the right passages?	Contextual Precision / Recall (RAG)
Did the agent finish the task?	Task Success Rate, Goal Achievement (Agent)
Did the agent stay in character?	Role Adherence (Agent)
Did the agent call the right tool with the right args?	Tool Correctness (Agent)
Does the response contain hateful language?	Toxicity, Bias (RAG)
Does the response leak PII?	PII Leakage (Security)
Does the response follow a structured format?	JSON Schema, Regex Match (Deterministic)
Something the built-ins don't cover ("does the answer keep our brand tone?", "did the agent gather all three info fields before booking?")	G-Eval or Custom Eval (Custom)

3. Add a security metric even in eval plans. PII Leakage, Prompt Injection Resistance, and Harmful Content are cheap to add on top of a correctness-focused plan and catch things a correctness metric never sees. Red-team is the deep pass; a couple of security metrics in a normal eval run is the smoke alarm.

The judge model matters

Every LLM-based metric routes through the plan's judge model (set on Step 3 of the wizard, plan-wide). Two rules that save pain later:

• Pick a strong judge and pin it. A weak judge (small model, low reasoning) produces noisier scores and misses subtle failures. GPT-4o / Claude-4-class as the judge is the practical sweet spot.
• Don't rotate the judge mid-experiment. Judge model is part of the measurement instrument. Change it and your pass-rate numbers stop being comparable across runs.

You can happily use a cheap model for the system under test and a stronger model for the judge. That combo gives the best signal-to-cost ratio.

Parameters and thresholds

Every metric carries a small set of parameters, rendered on wizard Step 5. Fields are typed (boolean, enum, number, string, string_list, text, JSON) and required ones are marked with *. Two parameters show up on almost every metric:

• threshold: the pass/fail bar. Catalog default is a sensible starting point; tune per plan.
• temperature: sampling temperature for the LLM judge. Higher values widen the verdict distribution (used by G-Eval's TCVA aggregation); lower is more reproducible.

Per-metric parameters are documented on each sub-page. When the catalog adds new parameters (library updates), they render generically at the bottom of the metric card in the wizard, no code change required.

Cost implications

Two things drive cost per run: the number of LLM calls per row and the size of the payload passed to each call.

• Deterministic metrics: zero LLM cost. Attach as many as you want.
• LLM-based metrics: one judge call per row per metric. A plan with three LLM metrics on a 300-row dataset is 900 judge calls plus the target's own 300 calls.
• G-Eval n_samples: G-Eval defaults to 20 samples per row for confidence scoring. That's 20 judge calls per row per G-Eval instance. Set it lower if cost matters more than confidence.
• Custom Eval consensus n_runs: same shape. Off by default; when turned on, multiplies judge calls by n_runs.

The Total cost KPI tile on the run detail page tells you the final number in USD to four decimals. Sanity check it after the first run of any new plan.

Iteration cadence

Metrics aren't set-and-forget. Reasonable rhythm:

• On every finished run: read the judge's reasoning on 3-5 failing rows for each metric. If the judge's "why" doesn't align with what you meant the metric to measure, tune the threshold or swap the metric.
• On big prompt / model changes: metric behaviour can shift under a new model. Re-check thresholds when you rotate the target model.
• Quarterly: prune metrics that consistently pass 100%. They cost money and give no signal.

Tips and pitfalls

• A metric that always passes is dead weight. Costs tokens, tells you nothing. Tighten the threshold or drop the metric.
• A metric that always fails is broken. Wrong columns, wrong threshold direction, or the metric doesn't match the plan's question. Read the judge's reasoning to figure out which.
• Don't stack four similar metrics. Answer Precision + Answer Relevancy + Contextual Recall + Faithfulness on the same plan largely ask the same question three different ways. Pick two that are actually different.
• Read the judge's reason column at least once per new metric. It's in the row expand view under Metrics. This is the fastest way to spot a rubric that isn't asking what you think.
• Save custom metrics as workspace presets once you've iterated on them a few times. See Custom metrics.
• Don't compare pass rates across runs with different metric sets. If Plan v1 had 3 metrics and Plan v2 has 4, the aggregate isn't comparable, per-metric numbers still are.

Dive deeper

• RAG and general LLM-judge metrics
• Agent metrics
• Security metrics
• Deterministic metrics
• Custom metrics

Related

• Test plans hub: where metrics attach.
• Datasets hub: design the row shape around the metrics you'll use.
• Custom metrics page: UI reference for the preset editor.