Custom metrics

Custom metrics exist for one reason: no fixed catalog can anticipate every question a team might want to ask about an AI system. "Does the answer keep our brand tone?", "Did the agent gather all three required fields before booking?", "Does the response follow our medical disclaimer template?", none of those fit a built-in metric neatly, but each is easy to describe in a sentence or two.

That sentence is the metric. Write it, pick a base, save it as a preset, attach to any plan. EvaliQA supports two custom-metric bases: G-Eval and Custom Eval. Both use an LLM judge to turn natural-language criteria into a 0..1 score. They differ in how the judgement is structured and how you tune them.

This page covers what a custom metric is, when to pick which base, how to write criteria that actually work, how the reusable presets flow works, and the traps to avoid.

Two ways to attach a custom metric

Custom metrics can be attached to a plan in two ways:

1. Inline in the wizard. On Step 4 (Metrics), tick G-Eval or Custom Eval under the Custom category and configure the params inline. The metric lives only on that plan.
2. As a workspace preset. Create the metric ahead of time in /custom-metrics, then attach it from Step 4 as a preset chip. Presets can be reused across every plan in the workspace and edited in one place.

Presets are the recommended path once you've iterated on a metric a few times. Every plan that uses the preset picks up updates automatically, so a rubric change propagates without hand-editing each plan.

G-Eval, when the rubric is a paragraph

G-Eval is the "give the judge a rubric and let it score" base. Best when what you're checking is fuzzy, subjective, or hard to break into discrete criteria.

Shape of the metric.

• criteria (text): a natural-language paragraph describing what the judge should score. This is the primary knob.
• evaluation_steps (string_list): optional. An ordered list of step-by-step instructions for the judge. When set, overrides criteria. Useful when the judgement benefits from an explicit chain-of-thought.
• n_samples (number, default 20): how many samples the judge draws per row. G-Eval uses log-probability weighting across samples to produce a smooth confidence-weighted score. Lower for cheaper runs; keep default for stable numbers.
• sampling_temperature (number, default 2.0): high on purpose, widens the sample distribution so the log-prob aggregation has variance to work with. Only touch this if you know what you're doing.
• threshold: pass/fail bar. Default 0.5.

Columns required. input, actual_output.

When to pick G-Eval.

• The rubric is a paragraph, not a list of discrete criteria.
• The judgement is holistic ("does this feel like our brand?", "is this answer respectful?").
• You want confidence-weighted scoring (each row gets a smooth 0..1 score, not a discrete pass/fail).

When to skip G-Eval.

• The rubric naturally breaks into several distinct criteria (use Custom Eval's verdict strategy for per-criterion verdicts).
• You need to reference dataset columns with {{placeholders}} (Custom Eval supports these; G-Eval does not).
• Cost is a concern, n_samples=20 per row is 20 judge calls per row per metric. Custom Eval defaults to one call per row.

Custom Eval, when you have discrete criteria

Custom Eval is the newer, more flexible base. Best when your rubric is a list of things the response should satisfy.

Shape of the metric.

• evaluation_criteria (string_list, required): one criterion per line. Each entry can contain {{placeholders}} that substitute dataset columns or connector output mappings. Example: The response must acknowledge the user's order number ({{order_id}}).
• strategy (enum, default verdict): scoring strategy. Two choices:
  • verdict: the judge scores each criterion separately (none, minor, partial, mostly, fully) and aggregates via TCVA (Threshold- Calibrated Verdict Aggregation). Fine-grained, backward-compatible default. Recommended for most cases.
  • direct: the judge returns one integer 0..10 for all criteria together, normalised to 0..1. Faster and coarser; the judge doesn't have to reason about each criterion separately.
• temperature (number, verdict-only): sampling temperature for the aggregation. Leave at default unless you're calibrating.
• n_runs (number, direct-only, requires consensus toggle): number of judge runs to average. Off by default (single call, cheap). Turn on the consensus toggle and set to 3 or 5 for more stable numbers.
• aggregation (enum, direct-only, requires consensus toggle): how to combine multiple runs (mean, median, min).
• threshold: pass/fail bar. Default 0.5.
• name: short label shown in the run results.

Columns required. input, actual_output.

When to pick Custom Eval.

• Your rubric is a list of discrete "must" and "should" statements.
• You want the judge's reasoning to name specific criteria that passed or failed (great for debugging).
• You need {{placeholder}} substitution to reference dataset columns or connector outputs in the criteria text.
• You want cheap-by-default: one judge call per row unless you turn on consensus.

When to skip Custom Eval.

• The judgement is a holistic feel, not a checklist. G-Eval handles fuzzy rubrics better.

The {{placeholder}} syntax

Custom Eval's evaluation_criteria are the only place in the platform where substitution happens. A criterion can reference:

• Connector output fields. Every custom output field on an active connector — its name in the connector's Response tab — is available as {{name}}, with nothing to configure on the test plan. Add confidence → metadata.score to the connector and {{confidence}} resolves on every run that uses it. This is the intended way to judge anything your API returns beyond the answer itself.
• Built-in test case fields. {{input}} and {{actual_output}} always; {{expected_output}} and {{retrieval_context}} only on rows that carry a value.
• Dataset columns, but only the ones you opt into: tick them under Extra fields when starting the run, and {{column_name}} resolves for that run.

{{tools_called}} is not available, despite the field existing on the test case — the judge never receives it.

Unresolved placeholders drop the criterion

This is the part worth internalising, because nothing about it is loud at run time. Before judging, the library keeps a criterion only if

1. it contains at least one {{placeholder}}, and
2. every placeholder in it resolves.

Everything else is dropped from the run and listed under skipped_criteria in the metric's verbose log. Two consequences:

• A criterion with a typo'd name ({{folow_up}}) is silently not judged — it is not passed through as literal text.
• A criterion written as plain prose with no placeholder at all is dropped too. A metric whose criteria are all prose scores 0.0 with the reason "No criteria could be evaluated".

Both forms show up as chips and warnings on the create form and in the plan wizard, so the fix is usually visible before the run.

G-Eval's criteria and evaluation_steps do not support placeholders at all. Its prompt interpolates input, actual output, expected output and retrieval context directly, and a {{name}} typed there reaches the judge as literal text. If you need substitution, use Custom Eval.

How to formulate criteria that actually work

The hardest part of a custom metric isn't the wizard, it's writing a rubric the judge can consistently score. A few patterns that work, learned the hard way.

Be specific about the pass condition

Vague: "The response should be helpful."

The judge will pass almost anything readable. "Helpful" isn't a threshold, it's a mood.

Better: "The response answers the user's question directly in the first sentence, then provides at most 2 supporting details, and ends with an actionable next step."

Now the judge has three concrete things to check.

Split into criteria, don't stack ANDs

Bad Custom Eval criterion: "The response is polite AND acknowledges the user's order AND provides a refund status AND does not mention competitors."

Split into four criteria:

• The response is polite.
• The response acknowledges the user's order number ().
• The response provides a refund status.
• The response does not mention competitors.

The judge scores each separately; you see which failed on which rows.

Anchor with examples

Judges score better when you show them what pass and fail look like. Add to evaluation_steps (G-Eval) or as a criterion (Custom Eval):

"Pass example: 'Your refund of $50 for order #12345 has been processed.' Fail example: 'Refunds usually take 5 days.'"

Explicit examples cut the judge's variance dramatically, often more than raising the sample count.

Name what "fully" vs "partially" means

For Custom Eval verdict strategy, the judge produces one of: none, minor, partial, mostly, fully. If you don't tell it what those levels mean per criterion, it improvises. Anchor them:

*"Criterion: The response provides a refund status.

• fully: gives the current status (processing, complete, refused) with a timestamp.
• mostly: gives the current status but no timestamp.
• partial: mentions the refund but doesn't say where it is.
• minor: mentions refunds vaguely.
• none: doesn't address refunds at all."*

Yes, this is verbose. Yes, it's worth it.

Reference the dataset, not abstractions

Vague criterion: "The response is factually correct."

The judge has no way to verify correctness on its own; it'll score based on plausibility.

Anchored criterion: "The response matches the reference answer: ."

Now the judge has a concrete comparison.

Test on 3-5 known-outcome rows before rolling out

Before you attach a new custom metric to a 500-row plan, run it on 5 hand-picked rows where you already know the expected verdict: 2 clear passes, 2 clear fails, 1 ambiguous. If the metric agrees with your verdicts, ship it. If it disagrees, tighten the criteria. Doing this manually with a Custom Eval preset takes 5 minutes and saves the run that would otherwise score everything as pass because the rubric was too loose.

Presets: the workspace-level flow

Save custom metrics as workspace presets when:

• You want to reuse the same rubric across many plans.
• Several team members should be able to attach the same metric without recreating it.
• You want to update the rubric in one place and have every plan pick it up.

Creating a preset. Go to /custom-metrics, click New custom metric. Sheet fields:

• Name (required): the label you'll see in the wizard's chip picker. Short and descriptive: "Brand tone", "Refund flow completeness", "Medical disclaimer present".
• Description: free-form notes on what the metric judges. Shown as a tooltip in the wizard.
• Base (required): G-Eval or Custom Eval.
• Threshold (0..1): pass/fail bar. Leave blank to use the base's default.
• Base-specific parameters: G-Eval's criteria / evaluation_ steps / n_samples / sampling_temperature, or Custom Eval's evaluation_criteria / strategy / temperature / (optional) n_runs / aggregation.

Save the preset. It appears as a chip under the Custom category in every test plan's Metrics step.

Editing a preset. Open the preset in /custom-metrics, edit fields, save. Plans that already bound this preset pick up the change on the next run. This is different from earlier versions of EvaliQA where the preset was copied inline. Bumping a preset is a workspace-wide change; treat it like a policy update, not a personal tweak.

Multi-instance metrics. Both G-Eval and Custom Eval can be attached to the same plan multiple times with different configurations. Use this to build a small suite of related rubrics ("brand tone check", "disclaimer present", "call-to-action included") that each score independently.

How to iterate on a custom metric

New custom metrics almost never work perfectly on the first try. A reasonable iteration loop:

1. Draft the rubric. Write 2-4 discrete criteria (Custom Eval) or a focused paragraph (G-Eval). Include one pass example and one fail example.
2. Test on 5 known-outcome rows. As above. Fast, cheap, catches the biggest rubric bugs.
3. Run on a small dataset (20-50 rows). Read the judge's reasoning on every failing row. Does the reasoning match your intuition?
4. Adjust and repeat. Common adjustments: tighten a vague criterion, add an anchor example, split a compound criterion into two, adjust threshold up or down based on where the distribution fell.
5. Save as a preset once stable. Now other plans and teammates can use it.
6. Revisit quarterly. Rubrics drift as products change. A rubric written for v1 of the product can be misaligned with v3.

When custom vs when built-in

Reach for a built-in metric when:

• The question fits a named metric in the catalog (correctness against a gold answer → Answer Precision; PII → PII Leakage; JSON validity → JSON Schema).
• You want low-cost, well-tested scoring behaviour.
• Reproducibility matters (built-ins have fewer moving parts).

Reach for G-Eval when:

• The rubric is holistic ("does this feel like our brand?", "is this response empathetic?").
• You want smooth confidence-weighted scores.
• Cost isn't the primary concern.

Reach for Custom Eval when:

• The rubric is a list of discrete criteria.
• You need {{placeholder}} substitution.
• You want cheap-by-default single-call scoring.
• You want the judge to name which criterion failed.

You can, and often should, use both bases together on the same plan: built-ins for the standard quality questions, one or two custom metrics for the product-specific ones.

Tips and pitfalls

• The judge is not omniscient. If the criterion asks "is this factually correct?", the judge scores plausibility, not truth. Pair with a reference-based metric or point the judge at the correct answer via {{expected_answer}}.
• Long criteria are fine; vague criteria are not. A 3-sentence criterion with concrete conditions scores more consistently than a 1-sentence criterion of the form "the response should be helpful".
• Cost creep with G-Eval. n_samples=20 per row per G-Eval metric adds up fast. On a 500-row plan with 2 G-Eval metrics that's 20,000 extra judge calls. Cap n_samples at 5-10 during iteration, raise for final release runs.
• Consensus in Custom Eval is worth it for high-stakes metrics. Turn on the consensus toggle for compliance / safety custom metrics where you want multi-judge agreement.
• Editing a preset is a workspace-wide change. Communicate before bumping a preset that other teams' plans depend on.
• Missing {{placeholder}} resolutions are silent — and costly. The criterion is dropped from the run, not passed through as text. Check skipped_criteria in the metric's verbose log after the first run rather than assuming every criterion was judged.
• Every criterion needs at least one placeholder. Prose criteria are skipped, and a metric made entirely of them scores 0.
• Verdict strategy is chattier in the run results. For each row you see per-criterion verdicts and reasoning. Great for debugging, worth the extra display space.
• Direct strategy is faster and coarser. Use for smoke-test runs where you just want a number, save verdict strategy for the runs where you'll read the reasoning.

Related

• Metrics hub
• Custom metrics page, UI reference.
• RAG and general LLM-judge metrics
• Agent metrics
• Security metrics
• Deterministic metrics