Know whether the change you just made was an improvement.

AI evaluation is the practice of measuring an AI system's output against test cases and metrics instead of judging a handful of answers by eye. You define what a good answer looks like, run the AI system over a dataset of representative inputs, score every answer, and repeat after each change. The result is a number you can defend: a pass rate per metric, tracked across releases, so you can tell whether a prompt change, a model swap or a new retrieval index made things better or worse rather than merely different.

In four steps. Build a dataset of representative inputs, each with an expected outcome or a rubric, including the awkward cases and the ones that should be refused. Choose metrics that test what actually matters for your product: correctness, faithfulness to retrieved context, tool use, tone, safety. Run the application through the interface you ship, usually an HTTP endpoint, so you are testing the real thing rather than a notebook. Then compare the run against the previous one and gate the release on the metrics that must not regress.

LLM-as-a-judge is using a language model to score another model's output against criteria you write, such as whether an answer is supported by the retrieved context or follows your refund policy. It exists because the qualities that matter most in a generated answer, faithfulness, completeness, tone, have no exact string to compare against, and human review stops scaling after a few hundred rows. A judge returns both a score and the reasoning behind it, so a failing row is something you can read and disagree with rather than a number you take on faith.

Reliable enough to gate a release, provided you treat it as a measuring instrument rather than an oracle. Three things decide it. Write criteria specific enough that two colleagues would grade a row the same way, because vague criteria, not weak models, cause most disagreement. Run the judge several times per row and take the majority or the median when the call is close. Use a deterministic check instead wherever the answer has a right shape, since a regular expression never drifts. Then calibrate: hand-label a sample, compare, and fix whichever of the two is wrong.

Start from the failure you are actually afraid of, then pick the cheapest metric that catches it. For answers built on retrieval, faithfulness catches invented claims and contextual recall catches a retriever that never returned the right passage. For agents, tool correctness checks the right call with the right arguments. Where the answer has a fixed shape, exact match, a JSON schema or a regular expression costs nothing and never drifts. Add a custom judge for the rules only your product has. EvaliQA ships 39 built-in metrics grouped by what you are testing.

Fewer to start than most teams expect. Fifty to a hundred well-chosen rows will surface most of what is broken, and at that size you can still read every failure by hand, which is worth more in the first week than coverage. Then grow the dataset from reality: every bug a user reports becomes a row, so it ends up shaped like your actual traffic rather than like what you imagined at the start. Spread matters more than volume. A hundred rows across the awkward cases beat a thousand paraphrases of the happy path.

Yes. Reference-free metrics score an answer on its own terms, which is what you need when there is no single correct response. Faithfulness checks that every claim is supported by the retrieved context. Answer relevancy checks that the answer addresses the question that was actually asked. A custom judge scores against a rubric you write rather than a fixed string. Deterministic checks still apply to shape: valid JSON, the expected language, a required disclaimer present. Expected outputs are worth writing where they genuinely exist, but they are not a precondition for starting.

Score retrieval and generation separately, or a failing run will not tell you which half broke. On the retrieval side, contextual recall asks whether the passage holding the answer came back at all, and contextual precision asks whether the useful chunks were ranked above the noise. On the generation side, faithfulness asks whether every claim in the answer is supported by the context that was retrieved, which is the check that catches a confident invention. Answer relevancy covers the remaining case, where the answer is true and grounded and about something else.

Grade the path, not just the final message, because an agent can report success in a fluent sentence after calling the wrong tool three times. Tool correctness checks the right tool with the right arguments, and a tools-error metric catches bad parameters, wrong call order and ignored results. Across a whole conversation, task success and goal achievement ask whether the user got what they came for, role adherence checks the agent stayed in character, and repetitive-pattern detection catches the loop where it tries the same thing forever.

Single-turn evaluation scores one input and one answer. Multi-turn evaluation scores a whole conversation: a user model with a persona and a goal drives the dialogue, and the exchange is judged as a whole. It matters because an AI system that answers any one question well can still lose the thread by the fourth turn, forget what the user said at the start, or drift out of character. The metrics differ accordingly: goal achievement, knowledge retention, role adherence and conversational flow have no meaning on a single exchange.

Yes, and that is where they earn their keep. Trigger a run from GitHub Actions, GitLab, Jenkins or CircleCI with an API token, and fail the build when a metric falls below the threshold set on it. The useful gate is comparative rather than absolute: the question is rarely whether the pass rate is high, it is whether this change made it worse than the run before. Because a run is frozen with every per-row score and the judge's reasoning attached, a failed build points at exactly which rows regressed and why.

Either your own key or platform credits, and it is a workspace setting rather than a per-run decision. Bring your own provider key and every judge call is billed to you by that provider, through any of 100+ providers, at a model you pick. Or run on platform credits, where there is no key to manage: the platform model does the judging and the workspace wallet is charged for it. Every run records the model that judged it, so a score from August is still readable in December after you have changed providers.