A regression gate for an AI agent has one job, which is to stop a pull request that makes the agent worse and let everything else through. Most teams who wire one up hit the same wall within a fortnight. The gate goes red on a change that touched nothing the agent does, someone reruns it, it goes green, and from that day on a red gate means "rerun it" rather than "read it". A gate that cries wolf is bypassed on exactly the urgent pull requests where it matters, and a bypassed gate is worse than no gate, because it gives the team a sense of coverage it does not have. The opposite failure is quieter. A gate set loose enough never to flap also never fires, and the regression it should have caught reaches production with a green check next to it.
The reason this is hard for AI agents and easy for ordinary software is that the measurement itself is noisy. Most of the metrics that say whether an agent did its job are scored by an LLM judge, a model that reads the transcript and produces a score, and that judge samples at a non-zero temperature. Two runs of the same test plan on the same dataset against the same unchanged agent do not produce identical numbers. On a 100-row plan with a strong judge, per-metric pass rates move by 2 to 3 percentage points between identical runs, the aggregate pass rate by 1 to 2 points, and 2 to 5 of every 100 rows change their verdict for no reason at all. A gate that fails on any drop is therefore a gate that fails at random. This post is about building the other kind, using EvaliQA's CI integration, a golden set and a threshold set with the noise floor in mind.
What counts as a regression
Before you can tell a regression from noise you need a definition narrow enough to test. For an AI agent, a regression is a row in the golden set that passed on the baseline run and fails on the candidate run, where the failure traces to the change under test. Every part of that sentence carries weight. "Row in the golden set" means a case someone chose on purpose, not a sampled production conversation. "Passed on the baseline" means there is a reference run against the same dataset version, with the same metrics and the same judge, so the two verdicts sit on the same scale. "Traces to the change" is the part that separates a regression from a judge flip, and it is the part only a person can confirm by reading the transcript and the judge's reasoning.
AI agent regressions arrive at three altitudes, and a useful gate watches all three. At the lowest level the agent calls the wrong tool or passes the wrong arguments, which Tool Correctness scores structurally against the row's expected tool calls without any judge. One level up the agent calls the right tools but misuses them, repeats a failing call or ignores a result, which is what Tools Error flags across its six error categories. At the top the agent completes every step and still does not do what the user asked, which Task Success Rate scores over the whole transcript. A prompt change that drops the "confirm before booking" instruction shows up as a sequence error at the middle altitude long before the aggregate pass rate notices, provided the metric is on the plan.
Noise is anything that flips a verdict without a change in the agent. Judge sampling is the obvious source but not the only one. A dataset that gained ten hard rows between runs drops the pass rate by definition, and that drop is coverage, not regression. A metric added to the plan gives every row one more way to fail. A judge model swapped on your own key puts the two runs on different rulers. A connector that timed out on four rows produces four errored rows, which leave the pass-rate denominator and can hide a real problem behind a green check. A gate that cannot tell these apart from a regression fires on all of them, and the team learns to ignore it.
Where the noise comes from
The LLM judge is the measurement instrument, and like any instrument it has a resolution below which differences are not readable. Most of EvaliQA's agent metrics route through the plan's judge model. On platform credits the judge is picked by the platform and kept fixed so runs stay comparable. On your own key you set it once on the test plan and it applies to every judge-based metric on that plan. The judge reads the transcript and produces a score between 0 and 1 with a short reasoning trace, and the metric's threshold turns that score into a pass or a fail. A row whose true quality sits near the threshold, say a Task Success Rate score that lands at 0.68 on one run and 0.72 on the next against a threshold of 0.7, flips between runs with nothing changed. The wobble is largest exactly where the gate is most sensitive.
The numbers in the introduction are what a strong judge gives you on a 100-row plan, roughly 2 to 3 points per metric, 1 to 2 points on the aggregate, and 95 to 98 percent of rows keeping their verdict. A weaker judge is noisier. A vague rubric is noisier. Task Success Rate adds its own variance when you leave task_description and success_criteria empty, because the judge then infers the goal and the criteria in extra calls, and it can infer them differently each time. Setting both fields explicitly removes those calls and makes the metric more reproducible, which is the first thing we do on any plan that feeds a gate.
The noise also compounds through the row verdict. A row passes only if every metric on the plan passes, so a plan with five judge-based metrics gives each row five independent chances to flip. Three metrics is a healthy count for a multi-turn plan, and a gate plan should sit at the low end of that. Every metric you add to the gate must earn its place by catching a failure class the others miss, not by being in the catalog.
Measure the floor before you set anything
You cannot reason about a noise floor you have not measured, and measuring it takes one extra run. Open the test plan, run it against the golden set with the connector the gate will use, and when it finishes run it again with nothing changed. On the plan's Eval runs tab, tick both runs and use Compare in the toolbar. The panel shows per-metric pass rates and the headline numbers side by side, as described in Comparing AI evaluation runs. Write down the aggregate difference, the largest per-metric difference and the number of rows that changed verdict. Those three numbers are the floor. If two identical runs on your 40-row set disagree on one row, your floor is one row, which on 40 rows is 2.5 points. If they disagree on three rows, the floor is 7.5 points and the plan is not ready to gate anything until you fix the metric or the rubric that is flipping.
Repeat the measurement whenever the instrument changes. A new judge model, a tightened rubric, a different threshold on a metric, or a dataset version that doubled in size all move the floor. The floor is a property of the plan plus the dataset plus the judge, not of the agent, which is why you measure it against an unchanged agent.
The golden set the gate runs
The gate runs against a golden dataset, the small hand-curated set of rows you trust as the source of truth, and the size of that set is a design choice with a direct effect on noise. A 30-row set that finishes in 40 seconds gets run on every pull request. A 300-row set that takes 6 minutes gets bypassed on urgent ones. The per-PR gate should be 20 to 50 rows of single-turn or short multi-turn cases that run in about a minute, with the full golden set and the long multi-turn sweeps on the nightly schedule. The resolution cost of a small set is real, since each row is 2 to 5 points of pass rate, and the threshold section below takes that into account.
Every row in the gate set needs the four things a golden row carries. A specific input, the exact phrasing a user would type. An expected outcome, which for an AI agent usually means expected tool calls on the row for Tool Correctness, plus either a reference answer or explicit success criteria. Metadata that survives the person who added it, at minimum a category, a difficulty and a source such as hand, production, ticket or incident. And a one-sentence reason to exist. Rows that duplicate another row with a slight rewording inflate the pass rate and belong in a metamorphic dataset instead. Trivial greetings belong in a smoke set. Bulk-generated rows that nobody reviewed belong in a living dataset, not in anything the gate reads.
The incident rows deserve their own treatment. Every bug that reached production becomes a permanent row tagged source = incident, and the pass rate on that subset should stay at 100 percent forever, because any drop means the same bug came back. An aggregate minimum of 0.9 will happily wave through a candidate that re-broke incident 142 while improving two unrelated rows. We keep the incident rows in a dataset of their own attached to the same plan and run a second gate step on it with the minimum set to 1.0. It is small, it is fast, and it is the only part of the gate where noise room is not allowed. If an incident row flips on judge noise, that row needs a deterministic check or an explicit rubric, not a looser threshold.
Version the set rather than editing it in place. Each run records the exact dataset id it used, so old baselines stay reproducible, but the sentence "the gate passes at 92 percent" stops meaning anything if the rows changed under it. Use Duplicate as new version for anything beyond a typo fix, point the gate snippet at the new version, and take a fresh baseline and a fresh noise measurement before you trust it. When the set grows, expect the pass rate to drop. Report it as coverage gained and reset the baseline, rather than reading it as a regression.
Metrics that flip for real reasons
The cheapest way to reduce noise is to score as much as possible without a judge. Deterministic metrics compute a verdict from a rule, give the same answer on the same output every time, and cost nothing. For an AI agent that means Tool Correctness first, which compares the tools the agent called and their arguments against the row's expected tool calls. Its evaluation_params lets you name the argument keys that carry intent and ignore the ones the agent fills by inference, such as a timestamp, so a legitimately different request id does not fail the row. check_ordering makes the sequence matter when it should, such as a confirmation email after a payment and not before. One default to change for a gate is exact_match, which is off, so partial matches earn partial credit and an agent that gets half its tool calls right can clear the default threshold of 0.5. Turn it on, or raise the threshold, so the metric means what the gate needs it to mean. Around it sit JSON Schema when the agent returns structured output, Contains in none mode for phrases the agent must never say, Non-Empty as a floor on every plan whose target can return nothing, and Length Check to catch a model update that doubles verbosity.
The judge-based layer then handles only what a rule cannot express. Task Success Rate with task_description and success_criteria filled in is the primary "did it work" signal, and Tools Error covers the misuse cases Tool Correctness cannot see. Resist adding Answer Relevancy, Faithfulness and a correctness rubric on top of each other to the same gate plan, because they ask overlapping questions three ways and each one is another chance for a row to flip. Semantic Similarity is tempting as a cheap judge-free correctness check, but know what it does not catch. With the default embedding model, "Refunds are available within 30 days" and "Refunds are not available within 30 days" score 0.93 against each other, so it is the wrong metric when correctness hinges on one fact.
For a verdict that must be stable because something hangs on it, a compliance rule or a safety check, a Custom Eval metric with consensus turned on trades cost for stability. With the direct strategy, enabling consensus runs the judge n_runs times and aggregates the scores, median by default. Three runs is a light consensus and five a strict one, and the cost is n_runs times the single call. Pair it with the Every criterion must pass option when one violated rule should fail the row. G-Eval already samples 20 times per row by default for its confidence scoring, which is why it is smoother and also why it costs 20 judge calls per row per metric.
Metric | Engine | Run-to-run stability | Cost per row | What it catches | What it misses |
|---|---|---|---|---|---|
Tool Correctness | No judge, structural comparison | Same verdict every run | Free | Wrong tool, wrong intent-bearing argument, wrong order | Right tool used badly, outcome quality |
JSON Schema, Contains, Non-Empty, Length Check | No judge, rule | Same verdict every run | Free | Broken format, forbidden phrases, empty or bloated answers | Anything about meaning |
Semantic Similarity | Embedding model, no judge | Same score every run | Free | Paraphrased correct answers | A negation that flips one fact |
Task Success Rate | LLM judge | Wobbles near the threshold, steadier with task_description and success_criteria set | 1 judge call | Whether the whole task got done | Partial completion under a lax inferred task description |
Tools Error | LLM judge | Wobbles near the threshold | 1 judge call | Six misuse categories across the transcript | Calling the wrong tool entirely |
Custom Eval with consensus | LLM judge, n_runs times | Stable at the cost of extra calls | 3 to 5 judge calls | Your domain rule, one criterion at a time | Rules you did not write down |
Set the minimum with noise room
The generated CI snippet fails the job on three conditions, and understanding them is the whole trick of setting the threshold. The job fails when the run ends in any status other than completed, when the pass rate is null because no row got a verdict, and when the pass rate is below EVALIX_MIN_PASS_RATE, which the snippet sets to 0.8 unless you change it. The pass rate is passed rows divided by passed plus failed rows, between 0 and 1, exactly as the run page shows it. Errored rows, the ones with no verdict because the connector failed or your AI system returned an HTTP error, are left out of that fraction and reported separately in errored_rows. Nothing in the script tells a regression from noise. The threshold has to.
[ "$STATUS" = "completed" ] || exit 1
[ "$PASS_RATE" != "null" ] || exit 1
awk -v r="$PASS_RATE" -v t="$EVALIX_MIN_PASS_RATE" 'BEGIN { exit !(r >= t) }'Set the minimum from the baseline and the floor, not from a round number. Take the baseline run's passed-row count, decide how many rows you are willing to let flip on noise before the gate fires, subtract, and divide by the row count. On a 40-row set where the baseline passes 37 rows and two identical runs disagreed on one, allowing one flip gives a minimum of 36 divided by 40, which is 0.9. A candidate that loses one row on judge noise lands at exactly 0.9 and passes, because the check is pass rate greater than or equal to the minimum. A candidate that loses two rows lands at 0.875 and fails, and two flipped rows on a measured floor of one is a signal worth reading. Allowing zero flips turns every noisy run into a red gate. Allowing three on a 40-row set means a change can break 7.5 points of coverage silently, so pick the number with the row count in front of you.
Leave the default 0.8 only if the arithmetic happens to land there. On the example above, 0.8 means 32 of 40 rows, five below baseline, and a change that breaks four golden cases merges green. The default exists so the snippet works on the first paste, not as a recommendation. The same arithmetic says a 20-row set cannot gate tightly at all, since one row is 5 points, and the honest fix is more rows rather than a cleverer threshold.
Add a fourth check the snippet does not include. The status response carries errored_rows, and a connector outage that errors half the set leaves the remaining half with a clean pass rate. We fail the job when errored rows are above zero on the gate set, because a run that scored 20 of 40 rows has not tested the agent. One more line after the pass-rate check does it.
ERRORED=$(printf '%s' "$RUN" | grep -o '"errored_rows":[0-9]*' | head -n 1 | cut -d ':' -f 2)
[ "$ERRORED" = "0" ] || exit 1Wire the gate into CI
The wiring itself is one HTTP call to start the run and a polling loop to read it, and the Integrations page generates both for you. The steps below assume the plan already has the golden set and the connector attached. Full request and response details are in the Integrations reference.
Two operational details trip people up in the first week. The trigger call returns 409 when a run is already in progress on the same dataset, with the id of that run in the response, and the snippet reads that as a failed start. Two pull requests opened a minute apart will hit this, so serialise the gate job in your CI or retry after the running one finishes. The second is the connector. PR runs should go against the staging connector, the agent as it will be if this change merges, while the nightly run goes against production so that drift from provider updates shows up there and not in someone's pull request. The AI evaluation pipeline page lays out the full cadence around the gate.
Read a red gate
When the gate fires, the job log ends with the run URL. Open it, then go to the test plan's Eval runs tab, tick the baseline run and the candidate, and use Compare. Read the per-metric deltas before the aggregate. A move of 5 points or more on one metric is a signal, and a consistent 2 points across several metrics is one too, while a single 2-point wobble on one metric is probably the floor you measured. Then go to the rows. Rows that passed on the baseline and fail on the candidate are the regression list, and three to five transcripts tell you more than any number. The judge's reasoning sits in the row's expand view under Metrics. If the reasons describe the change you made, such as the agent booking before it confirmed, the regression is real. If they describe nothing in particular, rerun the baseline once, and if the delta disappears it was noise, which you record as one more data point on the floor.
A red gate resolves in one of three ways. A real regression gets fixed, and if the failing case was not already in the golden set it becomes a row now. A row that flips repeatedly on noise is a bad row or a bad rubric, so tighten the success criteria, move the check to a deterministic metric, or rewrite the expected output, in a new dataset version. A dataset bug, where the expected output was wrong all along, gets corrected the same way. What you never do is lower the minimum to turn a red gate green. That is how a gate decays into a formality, and it happens one small concession at a time.
A per-PR gate on 40 rows is a controlled experiment, and its weakness is coverage. It tests the scenarios you wrote, so a failure class nobody has seen yet passes through until a production trace gets promoted into the golden set. It runs when a pull request opens, so a provider model update that lands on a Tuesday night is invisible until the nightly run against production reports it. Long multi-turn failures need length to show, which is why Knowledge Retention and Repetitive Pattern Detection belong on the nightly multi-turn sweep rather than a one-minute gate. Security regressions, a prompt injection defence that a prompt tweak weakened, are a red teaming question on a weekly cadence, not a correctness gate's job. And the CI status response carries no cost or latency, so a change that doubles the agent's token spend merges green unless someone reads the Total cost tile on the run page.
A CI gate for an AI agent stays useful only when a red result means something and a green one can be trusted, and both depend on treating the LLM judge as an instrument with a measurable resolution. Measure the noise floor with two identical runs before you set anything, and measure it again whenever the plan, the dataset version or the judge changes. Score everything you can deterministically, keep the judge-based layer to two or three metrics with explicit task descriptions and success criteria, and keep the gate set small enough to run in a minute. Set the minimum pass rate from the baseline minus the flips you are willing to tolerate, add the errored-rows check the snippet leaves out, and give incident rows their own step with no room at all. Then, when the gate fires, read the flipped rows rather than the number, and fix the row or the rubric rather than the threshold. The gate will miss what it was never built to see, new failure classes, overnight provider drift and multi-turn decay, which is why it is one rung of the pipeline and not the whole of it.
