Most teams building their first AI system naturally bring along the testing habits they already know. They write a set of test cases, check the output, connect everything to continuous integration, and watch the build go green. Then a customer asks the support assistant about returns, and it confidently describes a refund window the company has never offered. No test fails, because no test was designed to catch that kind of mistake. The assertions only checked that the reply was a non-empty string in the right format—and it was.
The team’s testing approach wasn’t the problem. Traditional software testing rests on two assumptions: the same input produces the same output, and someone can write down the correct output in advance. AI systems built on large language models challenge both assumptions. The same question may receive a different answer each time, and most questions have many acceptable answers as well as many subtly incorrect ones. That’s why AI systems benefit from a discipline that measures quality across a distribution of scores from evaluators rather than checking against a single expected value. This post explains what AI evaluation is, how it differs from the testing you know, and where your existing tests still have an important role.
What is AI evaluation?
AI evaluation is the systematic measurement of how well an AI system's outputs meet the quality you need, across a representative set of inputs, repeated whenever something changes. Each part of that definition matters. "Systematic" rules out trying a handful of prompts in a playground and deciding the answers look fine. "A representative set of inputs" means the test cases resemble what real users send, including the awkward ones. "Repeated whenever something changes" means evaluation is part of your release cycle, not a one-off audit before launch. The output is a set of scores, one per metric, that you can compare from one version of the system to the next.
Every evaluation has four parts. The dataset is the collection of test cases, for example, the inputs, plus, where possible, a reference answer or the context the answer should rely on. The metrics name the qualities you care about, such as whether an answer is faithful to the retrieved documents, whether it addresses the question, or whether a tool was called with the right arguments. The scorers turn each output into a number for each metric, using code, an LLM judge or a human reviewer. The test plan ties these together. It says which cases run, which metrics apply to which cases, how many times each case runs, and what score counts as good enough to ship.
It helps to separate model evaluation from system evaluation. Model evaluation compares LLMs on public benchmarks, and it answers a question about the model in general. System evaluation measures the thing you actually have to work, like your prompt, your retrieval pipeline, your tools and guardrails, and the model underneath, all working on your users' questions. A model that tops a leaderboard can still perform poorly inside your system, because the benchmark never saw your documents or your policies. When we write about AI evaluation on this blog, we almost always mean system evaluation.
The last thing to hold on to is the shape of the result. A test suite returns a list of passes and failures. An evaluation returns distributions: the average score on each metric, the share of cases above a threshold, the spread across repeated runs, and the individual cases that scored worst. This is not a cosmetic difference in reporting. It follows directly from how AI systems behave, and that is where the comparison with traditional testing starts.
How traditional testing works
Traditional software testing is built on assertions where for a given input, the code is expected to produce a specific output. For example, a unit test calls a function with known arguments and checks the returned value ,or an integration test that verifies that two components exchange the correct data. At every level, the pattern is similar, you need to set up an input, run the code, compare the result with the expected value, and report whether the test passes or fails.
This works because conventional code is deterministic. Given the same input and the same state, a function returns the same output every time, so a single passing run is strong evidence the function is correct for that input. It also works because the correct output is knowable. A function that calculates tax on an order has one right answer, and a developer can compute it by hand and put it in the test. Testers call the source of that right answer the oracle, and in most conventional software the oracle is the specification.
Determinism also gives failures a clear meaning, because when a test that passed yesterday fails today, something changed in the code, and the diff between the two commits tells you where to look. A flaky test, one that passes and fails on the same code, is treated as a defect in the test itself, something to fix or delete. Continuous integration depends on a red build meaning stop, and a green build meaning the change is safe as far as the tests reach.
Coverage has a clear meaning too. The behavior of conventional code is defined by its branches, so you can measure which lines and paths your tests exercise and find the gaps. A suite with high branch coverage has at least touched most of the decisions the program can make. It is decades of engineering practice, and it works very well for the systems it was designed for. The trouble starts when one of the components under test is a language model.
Why do AI outputs break assertions?
The first assumption to fall is determinism. A language model generates text one token at a time, sampling each from a probability distribution. Unless you remove the randomness, two runs of the same prompt can diverge from the first word. Even with the sampling temperature set to 0, many hosted model APIs do not guarantee identical outputs, because of how the provider batches and computes requests on its hardware. In a full AI system the variation compounds, because retrieval can return documents in a different order, and an agent that takes a different first step can end up somewhere else entirely. The consequence is that one run of a test case tells you what the system did once, not what it does.
The second assumption to fall is the oracle. Ask a support AI assistant "Can I return a jacket I bought last month?" and there are dozens of correct replies. They can be short or long, lead with the policy or with the steps, and apologise or not. An exact-match assertion rejects every one of them except the reply you wrote down. Loosening the assertion to check that the reply mentions the return window lets through replies that mention it and state it wrongly. String comparison cannot tell a correct paraphrase from a fluent error, and that distinction is exactly what you need to measure.
The third difference is that quality is not binary. An answer can be correct but ignore half the question. It can be faithful to the documents but written in a tone your brand would never use, or helpful except for one invented detail. Each of these is a separate property, and each comes in degrees. That is why evaluation uses several metrics instead of one assertion. Faithfulness measures whether the claims in an answer are supported by the retrieved context. Answer relevancy measures whether the answer addresses what was asked. Task-specific metrics cover things like correct refusals or valid tool arguments. A pass or fail verdict collapses all of this into a single bit and throws away the information you need to decide what to fix.
Taken together, these three differences help explain why evaluation works with distributions. When outputs vary, you need to sample them, when there are multiple correct answers, you need a scorer that evaluates meaning rather than simply matching strings, and when quality has several dimensions, each with its own range of possible degrees, scores are more useful than simple verdicts. An evaluation test case is therefore more like a small experiment than a single assertion: you run it several times, score each output using the relevant metrics, and examine how those scores are distributed.
Why coverage and regressions differ
Traditional testing measures coverage against the code, but the code does not define an AI system's behavior. The branching happens inside the model, in parameters you cannot instrument, and the input space is open-ended natural language. A support assistant built from a short prompt and a single API call has almost no branches to cover, yet users can ask it anything, in any language, with any amount of confusion or hostility. Code coverage for such a system can be close to complete while the behavior you care about is barely tested. A coverage report that says otherwise gives false comfort.
AI evaluation measures coverage against the input space instead. The useful questions are which user intents your dataset includes, which kinds of difficult input it represents, and which of the failures you have seen in production it reproduces. A good dataset also might include real traces from production, cases written on purpose to probe edge conditions, and adversarial inputs from red teaming, the practice of attacking the system deliberately to find unsafe or unintended behavior. You will never cover the input space the way you can cover branches. What you can do is make sure every category of use you know about is represented, then grow the dataset each time production shows you a category you missed.
Regressions change character too. In conventional software, behavior changes when the code changes, so every regression has a commit behind it. In an AI system, behavior can change while nothing in your repository does. The model provider updates the model behind the same name. A colleague edits a document in the knowledge base, and retrieval starts returning it for questions it does not answer. A small edit to the system prompt, made to fix one case, shifts the tone of every reply. Each of these changes the system, and none of them shows up in a code diff.
This has two practical consequences. First, evaluation has to run on more triggers than a code push. It should run when the model version changes, when a prompt changes, when the retrieval index changes, and on a schedule even when nothing seems to have changed. Second, a regression usually appears as a shift in a distribution rather than as a new red test. Average faithfulness drops a little. Or the share of correct refusals slips on one category of question while the overall score holds. Reading those shifts well means comparing runs case by case and category by category, not only watching a headline number.
How AI systems are scored
Once outputs are scored rather than asserted, the next question is who or what does the scoring. There are three kinds of scorer, and a working evaluation uses all three, each for the properties it measures well. Choosing between them is a trade-off between cost, speed and how much judgement a property requires. No single scorer covers everything, so the skill lies in matching each metric to the right scorer.
Deterministic checks are code, and they are the part of traditional testing that carries over most directly. They verify properties with a precise definition. The output parses as valid JSON, a required field is present, the reply contains a citation, a tool was called with arguments that match its schema, latency stayed under your limit. These checks are fast, free and perfectly repeatable, so use them wherever a property can be defined precisely. What they cannot do is judge meaning. A check can confirm that an answer cites a document, but not that the document supports what the answer says.
An LLM judge is a language model prompted to score another model's output against a rubric. You give it the question, the answer and, where relevant, the retrieved context or a reference answer, along with a clear description of what each score means. LLM judges make meaning-level metrics such as faithfulness and answer relevancy affordable across hundreds or thousands of cases. They also have known failure modes: a tendency to prefer longer answers, sensitivity to the order in which options are presented, and a preference for text written by models similar to themselves. A judge you have not checked against human labels is an unverified instrument, and you should treat its scores that way.
Human review is the reference the other two are measured against. A domain expert reading an answer can tell whether it is right in the way that matters to your users, which no rubric fully captures. Human review is also slow and expensive. Two reviewers often disagree, which is itself useful information about how well your metric is defined. Human labels are best spent where they have the most leverage: building and correcting the reference answers in your dataset, calibrating LLM judges on a shared sample, and reviewing the cases where automated scorers disagree or score lowest.
Scorer | Good for | Speed and cost | What it misses |
|---|---|---|---|
Deterministic check | Format, schema, citations present, latency | Instant and free | Whether the content is correct |
LLM judge | Faithfulness, answer relevancy, rubric criteria | Seconds per case, plus model API cost | Its own biases, until calibrated against humans |
Human review | Reference answers, calibration labels, worst cases | Slow and expensive | Scale, and reviewers disagree with each other |
In practice the layers stack. Deterministic checks run first and catch structural failures cheaply. LLM judges then score the meaning-level metrics on every case. Human reviewers look at a sample and at the worst cases, and their labels feed back into judge calibration. When you read an evaluation result, knowing which scorer produced each metric tells you how far to trust it and what it could have missed.
Where traditional testing still belongs
None of this means throwing away your test suite. An AI system is mostly ordinary software wrapped around one non-deterministic component, and you should test the ordinary software the ordinary way. The code that builds prompts, parses model output, validates tool arguments, retries failed calls, enforces rate limits and applies guardrail rules is deterministic. It has one correct behavior for each input, and unit tests are the right tool for it. Evaluating that code with an LLM judge would be slower, more expensive and less precise than a plain assertion.
The technique that makes this practical is to replace the model with a stub in those tests. A stub returns a fixed response that you choose. That lets you test how your code handles a well-formed answer, a malformed one, an empty one, a refusal and a timeout, all deterministically and at no cost. Many production incidents that look like model failures turn out to live in this layer. Think of a parser that breaks when the model adds a sentence before the JSON, or a retry loop that resends the same failing request until the budget runs out. The model behaved within its normal range, and the code around it did not handle that range.
The dividing line is simple to state. If a property has one correct value you can write down, test it with assertions. If it depends on what the model generated and several outputs could be acceptable, evaluate it with metrics. A tool-calling agent shows both sides. Whether your code rejects a tool call with a missing required argument is a unit test. Whether the agent chose the right tool for the user's request is an evaluation metric, scored across many cases.
The two also run at different moments. Unit tests run on every commit, finish in seconds, and block the merge when they fail. A full evaluation takes longer and costs money in model calls, so many teams run a smaller subset on every prompt or model change and the full dataset before a release. Both belong in your pipeline, but a green unit-test build tells you nothing about whether the answers got worse. Keep that distinction visible in your dashboards and your release checklist. Otherwise you risk a false sense of safety we see often: a team that ships because CI is green.
How to run your first evaluation
Setting up evaluation for the first time is less about tooling than about decisions. You have to decide what good means for your system, which inputs represent its real use, and what score is good enough to make a decision about different aspects of your AI system. The steps below are the order we follow with teams starting out. The running example is a customer support assistant that answers from a policy knowledge base.
Define what good means. Write down, in plain sentences, what a correct answer looks like for each main use case and which failures are unacceptable. For the support assistant: answers follow the policy documents, never invent terms, and hand off to a human when the question is about a specific order.
Build the dataset. Start from real user questions in logs or support tickets. Add cases for edge conditions and known past failures, then add adversarial inputs, and attach the relevant policy passage or a reference answer where you can. Go beyond the questions the demo was built around, because those are the inputs where the system is least likely to fail.
Choose metrics and scorers. Map each property from step 1 to a metric. Score faithfulness and answer relevancy with an LLM judge, handoff correctness with a deterministic check on the tool call, and format with code.
Calibrate the judge. Label a sample of outputs by hand and score the same sample with the LLM judge. Then adjust the rubric until the two agree well enough that you would act on the judge's scores.
Run a baseline. Run the current system on the full dataset, several times per case, and record the scores and their spread. Every future change is compared against this baseline.
Set thresholds and triggers. Set pass thresholds from the baseline and from what the business can accept. Decide which changes trigger a subset run and which trigger the full dataset.
Score live traffic. Score a sample of production traces with the same metrics, and turn every live failure you find into a new test case.
The example shows why the order matters. If you skip the first step, your metrics describe what is easy to measure rather than what your users need. The support assistant can then score well on relevancy while still inventing refund terms. If you skip calibration, a judge that is too lenient on faithfulness will report those invented terms as supported. And without a baseline, the first score you see has nothing to be compared with, so you cannot tell whether it is good or bad.
Resist the temptation to pick thresholds as round numbers before you have data. A threshold only makes sense relative to how the current system scores and how much those scores vary between runs. Suppose faithfulness moves by a few points between two runs of the same unchanged system. A threshold tighter than that will fail builds at random, and your team will learn to ignore it, which is how flaky tests die in conventional CI too. Measure the run-to-run spread first, then set the threshold outside it. The same spread tells you when a difference between two versions is real. A change that moves the average by less than the spread has not been shown to do anything.
The last step closes the loop that offline evaluation leaves open. Your dataset tests the scenarios you thought to write, while live traffic contains the ones you did not. Scoring production traces cannot tell you why a score moved, because the traffic itself shifts. It can, however, tell you that something needs a look, and each failure it surfaces makes the offline dataset more representative for the next release.
Next steps
If you have no evaluation in place today, start small and specific. Pick the use case of your AI system that would do the most damage if it went wrong, and write down in a few sentences what a correct answer looks like for it. Collect a few dozen real inputs for that use case from logs or tickets, and add the edge cases you already know about. That is enough for a first dataset, and it will teach you more than a large generic one borrowed from a benchmark.
Next, choose two metrics. Make one a deterministic check for a property you can define precisely. Make the other an LLM judge metric for the quality you care most about, which is usually faithfulness if the system answers from documents. Label a sample by hand, calibrate the judge against it, and run a baseline with several runs per case. Then attach the evaluation to the changes that can move it, starting with prompt and model changes. Keep your unit tests for the code around the model exactly where they are.
From there the work is incremental. Every production failure becomes a new case, every new use case gets its own metrics, and the thresholds tighten as you learn how much your scores vary. To set this up in EvaliQA, begin with a test plan for that first use case, with the dataset, the metrics and the thresholds in one place. That way the next change to your system is measured before your users measure it for you.