Deterministic metrics

The Deterministic category is a small but powerful set of metrics that do not use an LLM. Every deterministic metric computes its verdict from a rule you configure: a regex, a JSON schema, a length range, a keyword list. Same input, same output, every time.

Because there's no judge in the loop, these metrics are:

• Free. Zero token cost.
• Instant. Milliseconds per row, not seconds.
• Fully reproducible. Same rule on the same response gives the same verdict every run.
• Bounded in scope. They only check what the rule can express. For anything requiring semantic understanding, reach for a RAG or Custom metric.

You'll usually pair one or two deterministic metrics with a couple of LLM-based metrics on the same plan. The deterministic ones catch structural regressions cheaply; the LLM ones handle the fuzzy quality questions.

Regex Match

What it scores. Whether the response matches a Python regular expression.

Columns required. actual_output.

Default threshold. 0.5. Binary metric, threshold is basically "present or not".

When to use.

• Structured extraction tasks: does the response contain a valid ISO date, a phone number, a JSON block?
• Format guardrails: does the response start with a required prefix, end with a signature, use a required template?
• Redaction check: does the response NOT contain \d{16} (a bare card number)? Combine with full_match off and check the metric direction.

When not to use. Anything requiring understanding of meaning. Regex can check for the string "refund" appearing in a response but can't tell whether "we cannot process refunds" is a refusal or a confirmation.

Parameters.

• pattern (required): the regex. Python syntax.
• full_match: when true, uses re.fullmatch (the whole response must match). Default off, uses re.search (a match anywhere counts).

Tip. Test the regex on real production outputs before committing. re.search matching \d+ will pass any response containing any digit, that's usually not what you meant.

JSON Schema

What it scores. Whether the response validates against a JSON Schema document you provide.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use.

• Structured-output tasks where the model must return JSON. If your agent's downstream consumer parses the response as JSON, this metric catches malformed responses before they break the pipeline.
• Tool arguments, if the model is generating tool calls with structured args, validate the args here.
• Function-calling responses that need to match a specific schema.

When not to use. Free-form text responses. Also skip if your JSON schema is trivial (any object counts), the metric has nothing to enforce.

Parameters.

• schema (required): the JSON Schema document as JSON. Use draft 7 or later.

Tip. JSON Schema is more powerful than most people use it for. Set required arrays, additionalProperties: false, and enum constraints to catch not just "the JSON parses" but "the JSON has the right shape".

Length Check

What it scores. Whether the response length falls within a [min_length, max_length] range.

Columns required. actual_output.

Default threshold. 0.5. Binary.

When to use.

• Enforce response-size limits: a summariser that must produce 3-5 sentences, a Twitter-style bot capped at 280 chars, an email subject line under 100 chars.
• Catch verbosity regressions: if a model update starts producing responses twice as long, Length Check will flag it before you notice cost creep.
• Catch empty-response bugs: min_length = 1 catches empty strings fast.

When not to use. When length is genuinely open-ended. Also skip on plans where cost / latency KPIs already track this implicitly.

Parameters.

• min_length: lower bound. Default 0.
• max_length: upper bound. Default 10000.
• unit: chars (default) or words.

Contains

What it scores. Whether the response contains (or does not contain) specified keywords.

Columns required. actual_output.

Default threshold. 0.5.

When to use.

• Required-phrase checks: a compliance disclaimer, a required greeting, a call-to-action phrase that must appear.
• Forbidden-phrase checks: profanity lists, competitor names, banned URLs. Use mode: "none" for these.
• Coverage checks: a summariser that must mention all N topics from a source document, use mode: "all" and a keyword per topic.

When not to use. For semantic checks. "Response mentions our brand" is a Contains check; "response speaks favorably about our brand" is a G-Eval rubric.

Parameters.

• keywords (required): list of substrings.
• mode: any (default; pass if at least one keyword found), all (proportional score by coverage), none (pass only if NO keywords found).
• case_sensitive: default false.

Tip. For forbidden-phrase checks, remember to include lowercased variants, punctuation variants, and common typos. A single-word blocklist usually needs 3-5 entries per forbidden term to catch realistic attempts.

How to combine deterministic with LLM metrics

Deterministic metrics are cheap; use them liberally as structural floor. LLM metrics are expensive; use them for the fuzzy questions. A common pattern for a support-agent plan:

• Deterministic layer: JSON Schema (response format), Contains (required disclaimer present), Length Check (response between 50 and 500 chars).
• LLM layer: Answer Precision, Answer Relevancy, PII Leakage.

The deterministic layer runs in milliseconds and catches every structural regression. The LLM layer runs in seconds and catches every quality regression. Together you get fast fail-fast on structural issues without paying LLM cost per failure.

Tips and pitfalls

• Regex is trickier than it looks. Anchor your patterns (^ / $) when you mean "the whole thing", use non-greedy quantifiers (.*?) when you don't want the pattern to eat the rest of the response, and test against real outputs before shipping.
• JSON Schema doesn't validate free-text JSON snippets embedded in markdown. If your response is a mix of prose and a fenced code block containing JSON, JSON Schema tries to parse the whole response and fails. Add a preprocessing step (via Custom Eval or by cleaning the connector output) if you need to validate embedded JSON.
• Length Check counts what's in actual_output verbatim. Prompt scaffolding (system prompt, few-shot examples) doesn't count. What the model actually generated does.
• Contains + mode: "none" is a common way to build a cheap forbidden-phrase guardrail. Faster and more deterministic than an LLM policy check, at the cost of missing anything that isn't in your keyword list.
• Deterministic metrics don't produce a "reason" like LLM metrics do. The pass/fail is what you get. If you need diagnostic detail, add an LLM-based sibling metric that scores the same quality.

Related

• Metrics hub
• RAG and general LLM-judge metrics
• Custom metrics
• Evaluation, single-turn