Comparing runs

Comparing runs is how you answer "did the change help?". A prompt tweak, a model swap, a retrieval upgrade, a threshold change: none of them mean anything on their own. They mean something when you can put the new run next to the old one and see the delta.

The point of comparison is not to prove a change is better; it's to find out whether it's better, worse, or a wash on the axes you care about. And the point of this page is to make sure your comparison is actually measuring what you think.

Where the compare view lives

Not on the run detail page. The single-run page has no compare action. Compare lives on the test plan page:

1. Open /test-plans/{id} for the plan whose runs you want to compare.
2. Switch to the Eval runs tab. Every run of this plan lists here.
3. Tick two or more runs with the checkboxes.
4. Use the Compare action in the toolbar. The compare panel opens with per-metric pass rates, cost, and other headline numbers side by side.

You can compare runs from the same plan only. If you want to compare runs from different plans (e.g. two different metric configurations), export both CSVs and pivot in a notebook, or read the aggregate numbers on each run's page manually.

The four kinds of comparison

Different changes produce different signals. What to hold constant and what to look at depend on which of the four you're doing.

1. Prompt or model tweak (same plan, same dataset, same judge)

Setup. Same plan, same dataset version, same judge model, same target credential, target model may vary.

What to hold constant. Everything except the change under test. If you changed both the prompt and the model in the same run, the comparison tells you nothing about either one individually.

What to look at.

• Per-metric pass rate delta. Which metrics moved and how far? A 5+ pp improvement on the primary metric is a real win; a 2 pp is likely noise (see "Judge variance" below).
• Per-category delta. Did the change help everywhere, or did it help one category while hurting another? Trade-offs like this are common and worth naming explicitly.
• Cost delta. Longer prompts and heavier models cost more. A quality win that doubles cost isn't automatically a win.
• Latency delta. Same reasoning.
• Row-level agreement. Rows that passed in run A but fail in run B are your regression list. Rows that failed in A and pass in B are the wins. Read the transcripts on a handful of each; you learn more from three transcripts than from the aggregate.

2. Dataset refresh (same plan, new dataset version)

Setup. Same plan, dataset was updated between runs.

What to hold constant. Plan, model, judge. Only the dataset version changed.

What to look at.

• Aggregate pass rate is only comparable if the datasets are comparable. If you added 50 hard rows to the golden set, pass rate should drop. That's not a regression, it's a coverage expansion.
• Pass rate on the intersection of rows. If you added rows, compare only the rows that appear in both versions. This is fiddly today; export both CSVs and diff on row identity.
• New rows that fail immediately. These are the rows that were worth adding, and they tell you exactly where the current product has gaps.

Common trap. People treat dataset refresh runs like model-swap runs and freak out when pass rate drops. Growing the dataset temporarily drops pass rate almost by definition. Communicate the delta as "we tested 30 more difficult scenarios; here's the coverage gained", not "pass rate dropped".

3. Model swap (same plan, same dataset, new target model)

Setup. Plan and dataset unchanged; target model swapped.

What to hold constant. Plan, dataset, judge model. Especially: do not swap the judge model at the same time. If you do, the delta reflects both the target change and the measurement change, you cannot untangle them.

What to look at.

• Per-metric pass rate delta, same as prompt tweaks.
• Cost delta. Model swaps are often about cost efficiency.
• Latency delta. Model swaps often trade latency for quality.
• Behaviour on edge cases and adversarial inputs. Bigger models sometimes lose nuance under adversarial pressure; smaller models sometimes hold. Read the failing rows.
• Security metrics. Different models have different safety training. A swap that improves quality but hurts Jailbreak Resistance is a bad swap for a customer-facing product.

4. Metric or threshold change (compare with caution)

Setup. You changed a metric on the plan (added, removed, changed its threshold) between runs.

What to hold constant. Ideally nothing else. But since metrics are part of the plan, the plan itself changed.

What to look at.

• Per-metric numbers stay comparable. The metrics that didn't change scored the same rows the same way; those columns are still valid comparisons.
• Aggregate pass rate is not comparable. Adding a metric usually drops pass rate (more ways to fail); removing raises it. The aggregate number moved because you changed the yardstick, not because the product changed.
• Never use metric-change runs to argue that the product improved or regressed. They're plan-tuning runs, not product runs.

Judge variance: the hidden noise floor

Two runs of the same plan against the same dataset with the same target and the same judge will not produce identical numbers. LLM judges sample at non-zero temperature; small score differences on individual rows aggregate into small pass-rate wobble.

Typical noise floor on a 100-row plan with a strong judge:

• Per-metric pass rate: usually ±2-3 pp between identical runs.
• Aggregate pass rate: typically ±1-2 pp.
• Row-level agreement: 95-98% of rows keep the same verdict.

Two implications:

• A 2 pp delta between runs is probably noise. Rerun the older plan once to check whether the delta persists.
• You need a real effect size to claim a change helped. 5+ pp on the primary metric, or a consistent 2+ pp across several metrics, is a signal. A single 3 pp bump on one metric is possibly noise.

Reduce judge variance by using a stronger judge (larger, better- calibrated), lowering judge temperature, or turning on consensus aggregation on Custom Eval metrics (n_runs = 3-5).

What compare tells you well, and what it doesn't

Tells you well:

• Aggregate pass-rate direction of change (better / worse / wash).
• Per-metric direction of change.
• Cost and latency direction of change.
• Which specific rows flipped between runs.

Doesn't tell you:

• Why a metric moved. The compare view shows numbers; you still have to open the flipped rows and read the judge's reasoning to understand root cause.
• Whether the change is durable. One run is a snapshot; if the change involves new prompting logic that models will handle differently in 6 months, comparing today is limited.
• Anything about rows that weren't in both datasets. If the dataset changed between runs, aggregate compare is misleading.

The comparison workflow

The steps that produce useful comparisons every time:

1. State the hypothesis before the comparison. Write down what you expect the change to do. "The new prompt should improve Answer Precision by ~5 pp without hurting Toxicity or cost." Now compare and see if the numbers agree.
2. Change one thing at a time. If you changed the prompt and the retrieval and the model in the same run, the comparison is uninterpretable.
3. Check the noise floor first. If two identical runs disagree by 3 pp, a 2 pp "improvement" is nothing.
4. Read flipped rows. Pick 3-5 rows that went from pass to fail (regressions) and 3-5 rows that went from fail to pass (wins). Read the transcripts. The story is in the qualitative change, not the number.
5. Write down the delta in words. "Pass rate 78% → 84%, driven almost entirely by Contextual Recall on multilingual rows. Cost per row up 22% because of the longer context." This is what goes into the report.

Common comparison mistakes

• Comparing runs with different judges. The judge is the measurement instrument. Swap it and the numbers are on different scales.
• Comparing runs with different metric sets. Aggregate pass rate is meaningless when the yardstick changed. Per-metric numbers are still comparable for metrics that appear in both.
• Comparing runs with different dataset versions. Pass rate changed because the rows changed, not because the model changed.
• Chasing 1-3 pp wins. In the noise, in the noise, in the noise. Rerun before you celebrate.
• Comparing across models without pinning the judge. See above.
• Comparing partial (failed / cancelled) runs to complete ones. Different denominators, different populations.

Related

• Evaluation results: the hub.
• Finding bottlenecks: what to do once compare shows something moved.
• Preparing a report: how to communicate the comparison outcome.
• Test plans hub: why pinning the plan configuration matters for comparability.
• Metrics hub: judge model as measurement instrument.