Deterministic metrics

The Deterministic category is a set of metrics that do not use an LLM judge. Ten of them compute the verdict from a rule you configure: a regex, a JSON schema, a length range, a keyword list, a prefix, a format, a language. Two more, Semantic Similarity and Reference Match, compare meaning with a small embedding model that runs inside EvaliQA. Same input, same output, every time.

Because there's no judge in the loop, these metrics are:

• Free. Zero token cost. The embedding model runs locally, so it costs nothing per call either.
• Fast. Milliseconds per row for the rule checks, a little more for the embedding pair.
• Fully reproducible. Same rule on the same response gives the same verdict every run.
• Bounded in scope. They only check what the rule can express. For anything requiring judgement, reach for a RAG or Custom metric.

Every rule check scores 1 (pass) or 0 (fail), so the default threshold of 0.5 simply means "the rule held". The exceptions are Contains in all mode, which scores the share of keywords found, and the two embedding metrics, which score similarity from 0 to 1.

You'll usually pair one or two deterministic metrics with a couple of LLM-based metrics on the same plan. The deterministic ones catch structural regressions cheaply; the LLM ones handle the fuzzy quality questions.

Regex Match

What it scores. Whether the response matches a Python regular expression.

Columns required. actual_output.

Default threshold. 0.5. Binary metric, threshold is basically "present or not".

When to use.

• Structured extraction tasks: does the response contain a valid ISO date, an order number, a JSON block?
• Template checks that a fixed prefix or suffix can't express. For a plain prefix or suffix, Starts With and Ends With are simpler.
• Redaction checks: Regex Match passes on a match, so on its own it can't assert that something is absent. For a forbidden literal, use Contains in none mode. For a forbidden pattern, turn on full_match and use a negative lookahead, for example (?s)^(?!.*\d{16}).*$ passes only when no 16-digit run appears.

When not to use. Anything requiring understanding of meaning. Regex can check for the string "refund" appearing in a response but can't tell whether "we cannot process refunds" is a refusal or a confirmation.

Parameters.

• pattern (required): the regex. Python syntax.
• full_match: when true, uses re.fullmatch (the whole response must match). Default off, uses re.search (a match anywhere counts).

Tip. Test the regex on real production outputs before committing. re.search matching \d+ will pass any response containing any digit, that's usually not what you meant.

JSON Schema

What it scores. Whether the response is valid JSON that validates against a JSON Schema document you provide.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use.

• Structured-output tasks where the model must return JSON. If your agent's downstream consumer parses the response as JSON, this metric catches malformed responses before they break the pipeline.
• Tool arguments, if the model is generating tool calls with structured args, validate the args here.
• Function-calling responses that need to match a specific schema.

When not to use. Free-form text responses. Also skip if your JSON schema is trivial (any object counts), the metric has nothing to enforce.

Parameters.

• schema (required): the JSON Schema document as JSON. Use draft 7 or later.

Tip. JSON Schema is more powerful than most people use it for. Set required arrays, additionalProperties: false, and enum constraints to catch not just "the JSON parses" but "the JSON has the right shape".

Length Check

What it scores. Whether the response length falls within a [min_length, max_length] range, both ends included.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use.

• Enforce response-size limits: a summariser that must stay under 100 words, a Twitter-style bot capped at 280 chars, an email subject line under 100 chars.
• Catch verbosity regressions: if a model update starts producing responses twice as long, Length Check will flag it before you notice cost creep.

When not to use. When length is genuinely open-ended. To catch empty answers, Non-Empty is clearer than min_length = 1, which still passes a single space.

Parameters.

• min_length: lower bound. Default 0.
• max_length: upper bound. Default 10000.
• unit: chars (default) or words. Words are counted by splitting on whitespace.

Contains

What it scores. Whether the response contains (or does not contain) specified keywords.

Columns required. actual_output.

Default threshold. 0.5.

When to use.

• Required-phrase checks: a compliance disclaimer, a required greeting, a call-to-action phrase that must appear.
• Forbidden-phrase checks: profanity lists, competitor names, banned URLs. Use mode: "none" for these.
• Coverage checks: a summariser that must mention all N topics from a source document, use mode: "all" and a keyword per topic.

When not to use. For semantic checks. "Response mentions our brand" is a Contains check; "response speaks favorably about our brand" is a G-Eval rubric.

Parameters.

• keywords (required): list of substrings.
• mode: one of three.
  • any (default): scores 1 if at least one keyword is found.
  • all: scores the share of keywords found, so 3 of 4 scores 0.75.
  • none: scores 1 only if no keyword is found.
• case_sensitive: default false.

Tip. In all mode the default threshold of 0.5 passes a response that has half the keywords. Set the threshold to 1.0 when every keyword must appear.

Tip. Matching is by substring, not by whole word: the keyword cat also matches "category". For forbidden-phrase checks, include punctuation variants and common typos. A single-word blocklist usually needs 3-5 entries per forbidden term to catch realistic attempts.

Exact Match

What it scores. Whether the response is identical to the row's expected output.

Columns required. actual_output, expected_output.

Default threshold. 0.5. Binary.

When to use. Answers with exactly one right spelling: classification labels (refund, exchange, escalate), yes/no answers, IDs, codes, short extracted values.

When not to use. Any free-text answer. "Your order ships Monday." and "It ships on Monday." both fail against each other. Use Semantic Similarity or a judge metric for those.

Parameters.

• case_sensitive: default true. Turn it off when Refund and refund should count as the same answer.
• strip_whitespace: default true. Trims leading and trailing whitespace on both sides before comparing. Whitespace inside the text still has to match.

Starts With

What it scores. Whether the response starts with a given prefix.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use. Required openings: a greeting, a fixed label such as Answer:, a JSON response that must open with {, a classifier that must lead with its label.

When not to use. When the opening can vary in wording. Use Contains or a G-Eval rubric instead.

Parameters.

• prefix (required): the text the response must start with.
• case_sensitive: default true.

Tip. The response is not trimmed first. A leading space or blank line fails the check, so look at a few raw responses before you set the prefix.

Ends With

What it scores. Whether the response ends with a given suffix.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use. Required closings: a signature, a disclaimer that must come last, a fixed sign-off, a closing }.

When not to use. When the closing can vary in wording or punctuation.

Parameters.

• suffix (required): the text the response must end with.
• case_sensitive: default true.

Tip. As with Starts With, nothing is trimmed. A trailing newline or an extra full stop fails the check.

Non-Empty

What it scores. Whether the response has any text left once leading and trailing whitespace is removed.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use. As a floor on every plan whose target can return nothing: a connector that times out, a tool-only turn with no text, a model that stops early. Cheap enough to leave on everywhere.

When not to use. When an empty answer is correct for some rows, for example a filter that should return nothing.

Parameters. None besides the threshold.

Format Check

What it scores. Whether the whole response, trimmed, is a value of one named format.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use. Extraction tasks where the answer is a single value: "reply with the customer's email", "return the delivery date only".

When not to use. When the value sits inside a sentence. "Your email is a@b.co" fails, because the whole response has to be the value. Use Regex Match with re.search for that.

Parameters.

• format_type (required): one of
  • email: name@domain.tld.
  • phone: digits with an optional leading +, spaces, dashes, dots or slashes, and brackets around the first group only. +44 20 7946 0958 and (555) 123-4567 pass; +1 (555) 123-4567 does not.
  • date: numeric dates such as 2026-10-09, 2026/10/09 or 09/10/2026. Month names and dotted dates fail.
  • url: needs a scheme and a host, so https://example.com passes and example.com does not.

Tip. Format Check checks shape, not validity. A phone value as short as one digit passes, and 2026-13-45 passes as a date. Add a Regex Match when the exact shape matters.

Language Detection

What it scores. Whether the response is written in the language you expect.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use. Multilingual products: a German user should get a German answer, a support bot should not switch to English halfway through a thread, a translation step should produce the target language.

When not to use. Very short answers. One or two words, codes, numbers and names don't carry enough signal, and the detector may guess another language. An empty response always fails.

Parameters.

• expected_language (required): picked from a list of 55 languages, shown with their codes, such as en, de, pt, zh-cn, zh-tw. Only languages the detector can recognise are offered.

Tip. One metric checks one language on every row. For a multilingual dataset, run one plan per language.

Semantic Similarity

What it scores. How close the response is in meaning to the row's expected output. Both texts are turned into embeddings and the score is their cosine similarity, from 0 to 1.

Columns required. actual_output, expected_output.

Default threshold. 0.7.

When to use.

• Golden datasets where a correct answer can be phrased many ways and Exact Match is too strict.
• Cheap regression checks on large datasets: no judge calls, so a 10,000-row run costs nothing to score.
• Metamorphic pairs that should keep the same answer.

When not to use. When a small wording change flips the meaning. With the default model, "Refunds are available within 30 days." and "Refunds are not available within 30 days." score 0.93 against each other. For correctness that hinges on one fact, use a judge metric that names the fact, or Answer Precision when the answer is that fact alone.

Parameters.

• model_name: a sentence-transformers model id. Empty means all-MiniLM-L6-v2, which is already loaded. The model runs inside EvaliQA: no API key, no judge model, no cost per call.

Tip. Calibrate the threshold on your own data. Run the metric once on a handful of rows you know are right and a handful you know are wrong, and set the threshold between the two groups.

Reference Match

What it scores. How close the response is in meaning to a list of reference answers you write on the metric. Each reference is compared with the response the same way as Semantic Similarity, and the scores are combined into one.

Columns required. actual_output.

Default threshold. 0.7.

When to use. When every row should land near the same set of acceptable answers: a refusal that may be worded a few approved ways, a standard escalation message, an approved product description.

When not to use. When each row has its own right answer. The references are the same for every row; to compare against the row's own expected output, use Semantic Similarity.

Parameters.

• references (required): one reference answer per line.
• aggregation: max (default) passes if the response is close to any one reference; mean averages over all of them.
• model_name: same as Semantic Similarity.

How to combine deterministic with LLM metrics

Deterministic metrics are cheap; use them liberally as structural floor. LLM metrics are expensive; use them for the fuzzy questions. A common pattern for a support-agent plan:

• Deterministic layer: Non-Empty (an answer came back), JSON Schema (response format), Contains (required disclaimer present), Length Check (response between 50 and 500 chars).
• LLM layer: Answer Relevancy, Faithfulness, PII Leakage.

The deterministic layer runs in milliseconds and catches every structural regression. The LLM layer runs in seconds and catches every quality regression. Together you get fast fail-fast on structural issues without paying LLM cost per failure.

Tips and pitfalls

• Regex is trickier than it looks. Anchor your patterns (^ / $) when you mean "the whole thing", use non-greedy quantifiers (.*?) when you don't want the pattern to eat the rest of the response, and test against real outputs before shipping.
• JSON Schema doesn't validate free-text JSON snippets embedded in markdown. If your response is a mix of prose and a fenced code block containing JSON, JSON Schema tries to parse the whole response and fails. Add a preprocessing step (via Custom Eval or by cleaning the connector output) if you need to validate embedded JSON.
• Length Check counts what's in actual_output verbatim. Prompt scaffolding (system prompt, few-shot examples) doesn't count. What the model actually generated does.
• Contains + mode: "none" is a common way to build a cheap forbidden-phrase guardrail. Faster and more deterministic than an LLM policy check, at the cost of missing anything that isn't in your keyword list.
• Starts With, Ends With and Exact Match are case sensitive by default. Contains is not. Check the toggle when a check fails on a response that looks right.
• The reason is short. Deterministic metrics explain the verdict in one line, such as "Found 2/3 keywords (mode=all)" or the JSON validation error. For a written explanation of why an answer is wrong, add an LLM-based sibling metric that scores the same quality.

Related

• Metrics hub
• Metrics catalog
• RAG and general LLM-judge metrics
• Custom metrics
• Evaluation, single-turn