Most AI evaluation plans we see start in one of two bad places. Either every metric is an LLM judge, so a 300-row dataset with three metrics costs 900 judge calls per run and the pass rate wobbles by a few points between two runs of the same AI system, or every metric is an exact string comparison, so a correct answer in different words fails and the team stops trusting the run within a week. Both teams are trying to answer one question, whether the AI system gave a good answer, with a single kind of instrument. The problem this post deals with is how to split that question across three kinds of check, each used where it is strong, so that a run is cheap, reproducible wherever it can be, and still able to judge quality where nothing but a judge will do.
We call the arrangement the pyramid of AI evaluation. At the base sit deterministic metrics, rules that compute a verdict directly from the response and never call a model. In the middle sit semantic metrics, which compare meaning with a small embedding model but still involve no judge. At the top sits the LLM judge, a model that reads the response and reasons about it. The shape is the point. The base is wide because those checks are free and should run on every row of every run. The top is narrow because every judge call costs money, takes seconds and carries a little noise, so it should be spent only on the questions the lower layers cannot express.
The sections below take each layer in turn, name the EvaliQA metrics that live in it, and say what each catches and what it misses. They then show how the three layers combine on one test plan, how thresholds behave differently in each layer, where the whole structure stops being useful, and how the same idea carries into production monitoring. The running example is a customer support AI agent, because that is the AI system most teams evaluate first, but the structure applies to anything that returns text or structured output.
What the pyramid is
The pyramid orders metrics by how much judgement they need, and therefore by how much they cost and how far you can trust them to repeat. In EvaliQA a metric turns a response into a score, usually from 0 to 1, and a threshold turns that score into a verdict of passed, failed or errored. Every metric does this, whichever layer it belongs to, and the metrics hub describes the mechanic in full. What differs between layers is the engine behind the score. A rule check compares characters. An embedding metric compares vectors. A judge metric asks a model to read and decide. The pyramid says to use the cheapest engine that can express the check you want, and to move up only when the layer below cannot phrase the question.
The ordering also predicts how each layer fails, which is what makes it useful beyond cost. A deterministic rule fails loudly and honestly. It either held or it did not, and the reason is a single line such as "Found 2/3 keywords (mode=all)". An embedding metric fails quietly, by scoring two sentences as close when one word has flipped their meaning. A judge fails in the most human way, by scoring plausibility when you asked for truth, or by drifting between runs on an identical row. Knowing which kind of failure to expect from each layer is what lets you read a run rather than just collect its number.
The pyramid cuts across the way the EvaliQA wizard groups metrics. The Metrics step shows five categories, RAG, AI agent, Security and Safety, Deterministic and Custom, and groups by what the metric is for rather than by engine. Answer Precision sits in the RAG category but compares text overlap and makes no judge call, so by engine it belongs at the base. Tool Correctness sits in the AI agent category and compares the tools the AI agent called against the expected tools structurally, again with no judge. The Judge column in the metrics catalog is the reliable way to tell which layer a metric really lives in.
It helps to say plainly why a single layer is not enough. If everything is a judge, you pay model prices for structural checks a regular expression would settle in a millisecond, and when the pass rate drops 3 points you cannot tell whether your prompt changed or the judge did. If everything is a rule, you cannot evaluate a paraphrase, a tone of voice, or whether an answer is grounded in the passages that were retrieved for it. The pyramid is the answer to both problems at once, and the rest of the post is about where to draw the lines between the layers.
The deterministic base
The Deterministic category holds 12 metrics, ten of which are pure rule checks. They are Exact Match, Contains, Regex Match, Starts With, Ends With, Length Check, Non-Empty, JSON Schema, Format Check and Language Detection. Each scores 1 when the rule holds and 0 when it does not, so the default threshold of 0.5 simply means "the rule held". The one exception is Contains in all mode, which scores the share of keywords found, so 3 of 4 scores 0.75. These metrics cost no tokens, run in milliseconds per row, and give the same verdict on the same response every time. That last property is why they form the base. A deterministic failure is always a real change in the AI output, never noise in the instrument.
Their job is to be a structural floor. On the support AI agent, Non-Empty catches the connector that timed out and returned nothing, which otherwise looks like a quiet low score on every judge metric. JSON Schema catches a malformed response before the downstream parser breaks, and it does far more than check that the text parses if you set required arrays, additionalProperties to false and enum constraints. Contains in none mode is a cheap forbidden-phrase guardrail for competitor names or banned URLs, and in all mode it checks that a required disclaimer and a required greeting both appear. Length Check between 50 and 500 characters catches the model update that doubles response length before you notice it in the invoice. Language Detection confirms that a German user received a German answer, picked from 55 supported languages.
The base is honest about what it cannot do, and the docs are specific about it. Regex Match can find the word "refund" in a response but cannot tell whether "we cannot process refunds" is a refusal or a confirmation. A pattern like \d+ without anchors passes any response containing any digit, which is rarely what was meant. Format Check checks shape and not validity, so 2026-13-45 passes as a date. Contains matches substrings rather than whole words, so the keyword "cat" also matches "category", and a realistic blocklist needs 3 to 5 variants per forbidden term to catch typos and punctuation. Exact Match fails "Your order ships Monday." against "It ships on Monday." even though both are right. Each of these is a boundary of the layer, not a bug, and the fix is never to tune the rule harder but to move the question up a level.
A few metrics outside the Deterministic category belong at the base by engine. Answer Precision compares the response with the expected output by exact and normalised match, character and token overlap, and numbers with a tolerance. Its default threshold of 0.8 is strict for anything longer than a short phrase, which is why the docs keep it for numbers, names, codes, dates and yes or no answers. Tool Correctness compares the tools the AI agent called with the expected tools and their argument keys. Restricted Refusal checks the response against built-in and custom refusal phrases. None of them make a judge call, so attach them as freely as any rule check.
The semantic middle
The middle layer is two metrics, Semantic Similarity and Reference Match. Both turn text into embeddings with a small sentence-transformers model, all-MiniLM-L6-v2 by default, that runs inside EvaliQA, and both score the cosine similarity between two embeddings from 0 to 1 with a default threshold of 0.7. There is no API key, no judge model and no cost per call, and the same pair of texts gives the same score every run. That combination is what earns them a layer of their own. They read meaning, which no regular expression can, but they are still deterministic, so a drop in Semantic Similarity between two runs is a real change in what the AI system said.
Semantic Similarity compares the response with the row's own expected output. It is the right correctness metric for a golden dataset where a correct answer can be phrased many ways and Exact Match is too strict, and because scoring is free it is the natural regression check on large datasets, where a 10,000-row run costs nothing to score. It also fits metamorphic pairs, where a rephrased question should keep the same answer. Reference Match compares the response with a list of reference answers written on the metric itself, the same list for every row. That suits a refusal that may be worded a few approved ways, a standard escalation message or an approved product description. Its max aggregation passes if the response is close to any one reference, while mean averages over all of them.
The weakness of this layer is negation, and the docs give the exact number. With the default model, "Refunds are available within 30 days." and "Refunds are not available within 30 days." score 0.93 against each other, comfortably above the 0.7 threshold. Embeddings measure how close two texts are in topic and phrasing, not whether they are true in the same way. Any answer whose correctness hinges on one fact, a date, a quantity, a yes where a no was expected, cannot be trusted to this layer. When the answer is that fact alone, Answer Precision at the base is the better choice. When the fact sits inside a sentence, you need a judge that is told which fact to check. Very short responses are a second limit, since a one-word answer carries too little signal for the embedding to mean much.
The LLM judge at the top
A judge metric sends the row's input, the AI output and any reference material to the plan's judge model and asks for a score with a short reasoning trace. Most of the built-in catalog is this shape. Everything in RAG except Answer Precision, most of the AI agent category, most of Security and Safety, and both Custom bases, G-Eval and Custom Eval. Each one is one judge call per row per metric, so a plan with three judge metrics on a 300-row dataset is 900 judge calls on top of the 300 calls to the AI system itself. And the same judge scoring the same row twice can return slightly different numbers. That is the price of the top of the pyramid, and it is worth paying only for questions the lower layers cannot phrase.
Those questions are real and there are many of them. Faithfulness checks whether every claim in the response is supported by the retrieval context, with a default threshold of 0.7 that the docs suggest raising to 0.9 or 0.95 for medical, legal or financial products where any ungrounded claim is a bug. No rule can do that, because the claim and the passage that supports it never share a fixed string. Answer Relevancy scores whether the response addresses the question that was asked, needs no expected output at all, and is the metric of choice when you have no gold answer. Role Adherence and Goal Achievement read a whole conversation. G-Eval and Custom Eval take a rubric in plain language, such as whether the answer keeps your brand tone or whether the AI agent gathered all three required fields before booking, and score it.
Because this layer is expensive, its parameters are mostly about spending less. G-Eval draws 20 samples per row by default for confidence-weighted scoring, which is 20 judge calls per row per metric. On a 500-row plan with two G-Eval metrics that is 20,000 extra judge calls, so the docs recommend capping samples at 5 to 10 while iterating and raising them for release runs. Custom Eval makes one call per row by default and only multiplies that when you turn on consensus with 3 or 5 runs, which is worth it for compliance or safety rubrics. The judge model itself is part of the instrument. On your own key you pick it once on the wizard's Step 3 and it applies to every judge metric on the plan. On platform credits the platform picks it and keeps it fixed so runs stay comparable. Either way, changing the judge mid-experiment makes pass rates across runs meaningless.
The judge's blind spot is that it scores plausibility when it has nothing to check against. A Custom Eval criterion that says the response is factually correct gives the judge no way to verify anything, so it rewards answers that sound right. The fix is to anchor the criterion to data, for example by referencing {{expected_output}}, which Custom Eval substitutes from the row. A criterion that says the response should be helpful passes almost anything readable, because helpful is a mood rather than a threshold. The docs' discipline for this layer is to test any new rubric on 5 hand-picked rows, 2 clear passes, 2 clear fails and 1 ambiguous case, before attaching it to a 500-row plan, and then to read the judge's reasoning on 3 to 5 failing rows after every run. If the reasoning does not match what you meant the metric to measure, the rubric is wrong, not the AI system.
How the layers work together
The docs give a pattern for a support AI agent plan that is a pyramid in miniature. The deterministic layer is Non-Empty, JSON Schema for the response format, Contains for the required disclaimer and Length Check between 50 and 500 characters. The judge layer is Answer Relevancy, Faithfulness and PII Leakage. Adding Semantic Similarity against the expected output for the golden rows completes the three layers. A row's overall verdict is the AND of every metric on the plan, so one failing rule fails the row, but the per-metric pass rates in the run's KPI tiles tell you which layer fell over. The deterministic layer finishes in milliseconds and costs nothing, the semantic layer costs nothing either, and only the three judge metrics appear in the Total cost tile.
The value of the stack shows when the layers disagree, because each disagreement names a specific kind of failure. Semantic Similarity high while Faithfulness is low means the answer resembles the gold answer but is not supported by the retrieved passages, which usually means the model answered from memory or the gold answer was never in the context to begin with. Semantic Similarity low while Answer Relevancy is high means the AI system answered the question but not with the gold answer, and you should read the rows before deciding whether the gold answer is the only acceptable one. JSON Schema failing while every judge metric passes means the judge read the prose happily and the downstream parser would have crashed. Every base metric passing while a judge metric fails is the most common real regression, the shape is fine and the content is wrong. A single aggregate pass rate hides all four patterns.
The stack also disciplines how many metrics you attach. Start with one or two per layer that answer the specific question you have, not eight because the catalog offered them. A correctness metric plus Answer Relevancy plus Contextual Recall plus Faithfulness largely ask the same question four ways and quadruple the judge bill. Pass rates across runs with different metric sets are not comparable either, because the AND of five metrics is a different quantity from the AND of six, so once a plan is in use add metrics deliberately and compare per-metric numbers across the change.
Layer | Example metrics | Cost per row | Reproducible | Catches | Misses |
|---|---|---|---|---|---|
Deterministic | JSON Schema, Contains, Length Check, Non-Empty, Exact Match, Answer Precision | Zero, milliseconds | Always | Structural regressions, missing phrases, format and length drift | Anything that needs meaning, a refusal versus a confirmation |
Semantic | Semantic Similarity, Reference Match | Zero, local embedding model | Always | Paraphrased correct answers, drift from the gold answer at scale | Negation and single-fact flips, which can score 0.93 |
LLM judge | Faithfulness, Answer Relevancy, Role Adherence, G-Eval, Custom Eval | One or more judge calls per metric | Slight variance between runs | Grounding, relevance, tone, task completion, rubrics in plain language | Truth without a reference, vague rubrics, a rotated judge model |
Setting thresholds per layer
Thresholds behave differently at each level of the pyramid, and treating them alike is a common way to get a plan that passes everything. At the base the rule checks score only 0 or 1, so the default threshold of 0.5 is not a knob at all. The one setting that matters is Contains in all mode, where the default 0.5 passes a response that has half the required keywords. Set it to 1.0 when every keyword must appear. The other base decisions are the case sensitivity toggles, since Starts With, Ends With and Exact Match are case sensitive by default while Contains is not, and a check that fails on a response that looks right is usually one of those toggles.
In the middle layer the threshold is a measurement you take, not a default you accept. The 0.7 default on Semantic Similarity and Reference Match is a reasonable first guess for the default embedding model, but the gap between your correct and incorrect answers depends on how long and how varied they are, which is why the calibration in the tip above matters. Once calibrated it should stay fixed for as long as the embedding model does, because the metric is deterministic and any later movement is the AI system.
At the top, defaults range from 0.6 on Answer Relevancy to 0.8 on Bias, and the docs are explicit that tuning them is legitimate. If a judge metric passes 100 percent of your golden set on the first run, the threshold is almost certainly too low to discriminate, and tightening it, say from 0.7 to 0.85, then re-running is the correct response. Four security metrics invert the direction. Prompt Injection Detection, Jailbreak Detection, PII Leakage and Harmful Content score risk, so lower is safer, they pass below the threshold, and raising their threshold makes them more lenient rather than stricter. The two Resistance metrics and Policy Compliance pass only on a score of exactly 1, whatever the threshold says.
Two housekeeping rules apply across all three layers. Errored is a third verdict, produced when a metric crashes on a bad response format, a provider timeout or a missing column, and errored rows are excluded from the pass-rate denominator, so a run with many errors can show a flattering pass rate. Filter by the Errors chip after any new plan. And a metric that passes every row for a quarter is dead weight, it costs tokens if it is a judge and tells you nothing in any layer, so tighten it or remove it.
What the pyramid does not catch
The pyramid structures how you score a dataset. It says nothing about what is in the dataset, and a perfectly layered plan on 40 happy-path rows will pass a broken AI system. The attack you did not think to write is not found by any of these metrics, and the red-team modes in EvaliQA do not use attached metrics at all, they score every row per vulnerability and attack technique with their own judges. A couple of security metrics on a normal evaluation plan are a smoke alarm, and the docs describe them that way. Red teaming is the deep pass and a separate exercise.
The base and the middle only read the response. Every rule check and both embedding metrics take actual_output and, at most, expected_output, which means a failure that lives in the shape of a conversation has no cheap equivalent. An AI agent that forgets a fact the user gave three turns earlier is caught by Knowledge Retention, a judge metric over the whole dialogue, and by nothing lower in the pyramid. In multi-turn plans the top layer carries more of the weight, and the cost budget should expect that.
The judge layer has its own gaps that are easy to miss. Faithfulness only sees the retrieval context that reaches it, through the connector's Retrieval context path or the dataset's retrieval_context column. If your pipeline passes retrieved passages inside the system prompt, Faithfulness scores the answer as if there were no grounding at all, and on a row with no context it shows n/a and makes no judge call. Custom Eval drops any criterion whose placeholder does not resolve, silently, and a criterion written with no placeholder is dropped too, so a typo in {{folow_up}} means that criterion was never judged. The pyramid lowers how often you need to read the judge's reasoning. It does not remove the need.
The pyramid in production
The same shape appears in online evaluation, where live sessions are scored as they arrive instead of in a test run. The Runtime auto-eval panel has three switches, and they are the three layers. Deterministic detectors are sub-second checks that need no model, looking for explicit errors, retry loops, redundant tool calls, and cost and duration outliers. They cost nothing and the docs recommend leaving them on always. Bound metrics are scored per trace with one judge call per trace per metric, and the Rule scope on each binding lets you skip the traces that do not need judging, such as health checks or greetings. The AI Analyst is a strong model that reads the whole session, the detector findings and the metric verdicts, and writes recommendations, one call per session taking about 30 seconds.
Reading it as a pyramid gives you the same spending rules as offline. Let the free base run on every session. Scope the judge metrics to the traces you would act on, and prefer conversation metrics for chat products because one call per session is cheaper than one per turn. Leave the AI Analyst off until someone is reading its findings and following them up, because its value is entirely in the recommendations and not in the fact that it ran. Production also has a limit that offline does not. Traffic shifts, so a moving pass rate can reflect your users as much as your AI system, which is why the controlled test plan remains the place where you decide whether a change is good.
Summary
The pyramid of AI evaluation is a rule about where to spend judgement. Deterministic metrics are free, instant and perfectly reproducible, so they run on every row and catch every structural regression. Semantic Similarity and Reference Match add meaning without adding cost or noise, so they carry correctness checks on golden datasets and large regression runs, as long as you remember they cannot see a negation. LLM judge metrics are the only layer that can score grounding, relevance, tone and task completion, so they are reserved for exactly those questions, pinned to one judge model and kept to the few metrics that answer the question you actually have.
What you get from stacking them is not just a cheaper run but a readable one. When the base passes and the judge fails, the content is wrong. When the semantic layer passes and Faithfulness fails, the answer came from somewhere other than the retrieved context. When the schema check fails and the judge is happy, the parser downstream would have broken. The pyramid does not tell you what to put in the dataset, and it does not relieve you of reading the judge's reasoning, but it turns one blurred pass rate into three layers of evidence that each say something specific about your AI system.
