Evaluation results

An eval run is one execution of one test plan over one dataset. Once it finishes, it's frozen: every metric score, judge reasoning, latency, and token count is preserved. The run itself is not the deliverable, what matters is what you learn from it and what you do next.

This page is the methodology of that learning step: how to read a run without missing the important signal, how to find where the system is actually failing, and how to hand a summary to someone who won't read the raw rows. The three subpages go deeper into specific analysis patterns.

The analysis workflow

Every completed run gets the same treatment. Not every step takes long, but every step is worth doing before you form an opinion.

1. Skim the KPIs. Rows / Pass rate / Avg latency / Total cost. Sanity check: did the run actually run to completion? Is the cost in the ballpark you expected? Ninety seconds tops.
2. Read the per-metric bars. Which metric is dragging the pass rate down? A run at 65% pass rate can mean "one metric at 20% and three at 90%" or "everything at 65%". Different diagnoses.
3. Filter to Failed rows. Read the judge's reasoning on 5-10 rows. Look for patterns. Root causes cluster; a single-row failure is rarely a single-bug story.
4. Categorize the failures. Which category / tag / persona / mode fails the most? This is where the bottleneck analysis lives.
5. Compare to the last run. Did this change help, hurt, or move things sideways? See comparing runs.
6. Write the summary. For yourself, your team, or your PM. See preparing a report.
7. Decide the next iteration. New rows to promote, a metric to tune, a prompt to change, a model to swap.

Skipping steps 3-5 is how teams end up chasing the wrong bug. The aggregate pass rate hides the story; the story lives in the rows.

The five run statuses

A run is always in exactly one state:

• Queued (grey badge). Waiting for a worker slot. Nothing to analyze yet.
• Running (brand-blue badge). Rows are being processed. The page polls every 2 seconds; the progress bar and header update live. KPI tiles and per-metric charts render only after the run finishes, so there's not much to read yet, only the progress and any errors on the first few rows.
• Completed (green badge). Every row got a verdict (pass, fail, or error). This is the state you analyze.
• Failed (red badge). A systemic error prevented completion (a credential became invalid mid-run, the worker crashed, the provider had an outage). Any rows that got a verdict before the failure are still visible; the rest are missing.
• Cancelled (grey badge). Someone clicked Cancel on the header. Partial results are preserved.

Only completed runs give you a fair aggregate. Partial (failed / cancelled) runs can still be worth investigating for the rows they did score, but the pass rate is not comparable to a full run.

What a completed run stores

Read this once, understand what you can and can't ask of a run.

• Plan snapshot at run time. Mode, model, judge model, metrics with their thresholds, connector configuration. Editing the plan after the run does not affect this snapshot. Old runs stay reproducible.
• Dataset snapshot at run time. The exact rows the run scored, frozen. If you regenerate the dataset later, old runs still point at the version they used.
• Per-row results. For every row × metric:
  • Score (0..1) and verdict (passed / failed / errored).
  • Judge reasoning text (for LLM-based metrics).
  • Retrieval context (if the plan uses RAG).
  • Tools called (if agent mode).
  • Response time (ms) and HTTP status.
  • Token usage (input / output / total) and cost (USD).
• Reports generated from the run. Zero, one, or many. Linked in the header.

This snapshotting matters: old runs stay authoritative sources of what happened, even if the plan or dataset has moved on since.

Reading the run detail page

Top to bottom, what you see on /eval-runs/{id} once the run is completed:

Header. Run short-id, mode, model, and status badge. Actions in the top right:

• Cancel (destructive red) appears while the run is queued or running. Fires immediately, no confirmation dialog. Cancelling a run is not recoverable, use with intent.
• Delete replaces Cancel once the run is inactive. Asks for confirmation ("Delete this eval run? All per-row results will be removed. This cannot be undone.").
• CSV downloads every row + verdict as a CSV file.
• Open report or Generate report / New report manages the human-readable summary. Disabled while queued or running. See preparing a report.

Four KPI tiles (only rendered when status is completed):

• Rows. Total scored.
• Pass rate. Mean pass rate across metrics, coloured green at 80% or higher, amber at 50-80%, red below.
• Avg latency. Computed from the currently-loaded page of rows. Not p95, not p99, just the mean of what you can see. For the full distribution, drill into rows manually.
• Total cost. USD to four decimals, aggregated from the target's tokens across all rows (does not include judge cost, which is charged to your judge credential separately).

Per-metric charts. A tab component below the KPIs, one tab per metric with the score distribution and threshold marker. Red-team runs get an additional summary block (one horizontal bar per vulnerability with breach counts and percentages).

Per-row accordion. Header row: #, Status, Input, Actual output, Latency. Filter chips above: All / Passed / Failed / Errors, each with a count. Filtering is client-side over the current page (page size 25).

Row expand (click the chevron). Two-column grid:

• Left: Test case #N and launch time. For single-turn: Input, Actual output (markdown-rendered), Expected answer, Context chunks (numbered, clamped at 300px with "See all"), System prompt. For multi-turn: the full conversation with user right / assistant left.
• Right: one Metrics card per metric (name, pass/fail chip, score / threshold, colored bar with threshold marker, judge reasoning text, per-metric cost). Clicking the metric name opens the verbose log sheet with the full judge prompt and response. Below that: Response time (ms + HTTP status), Token usage (model, input / output / total, cost).
• Full-width below (agent / red-team modes): Tools called chip list, Custom outputs field blocks.

The judge reasoning text is where most of your analysis time goes. It's the model's explanation of why it gave the score it did. Reading it is fast (a few sentences per row) and it tells you whether the metric is asking what you thought it was.

Which KPI should you fix first?

A common trap is to react to whichever number looks worst. Better order of triage:

1. Errors first. Rows with errored metrics don't count in pass rate, but they mean the metric or the target crashed for that row. Systemic errors (5+ same error) point at a bug in the plan (wrong column, missing credential, connector broken).
2. Then failed rows. Read the judge's reasoning on 5-10 fails. Categorize the root cause, do not fix individual rows.
3. Then latency, if any percentile is outside your SLA.
4. Then cost, if it's growing unexpectedly or breaking budget.
5. Only then aggregate pass rate, and only as a communication number for the report.

Aggregate pass rate is a symptom. Per-row reasoning is the diagnosis.

The three deeper analyses

For each of the three main "what do I do now?" questions, there's a subpage:

• Finding bottlenecks: methodology for locating where quality, cost, or latency is actually failing. Slicing by category, persona, mode, and metric.
• Comparing runs: how to compare two runs (model swap, prompt tweak, dataset refresh) so the delta means something. Where the compare view lives, what to look at, common ways teams read comparisons wrong.
• Preparing a report: turning a completed run into a document you can hand to a PM or a stakeholder. What the AI-generated report gives you, what you should add or edit, what to leave out.

Tips and pitfalls

• The aggregate number is for reporting, not diagnosis. Never form an opinion from the pass-rate tile alone. Read the rows.
• A run scored below 50% is a signal to check the setup, not necessarily to iterate on the product. Wrong metrics, wrong dataset, wrong judge, wrong threshold direction, any of those produces a suspiciously low number.
• Cancel is instant, no undo. Read the confirmation state twice before clicking. Cancelled runs preserve their partial rows but the aggregate is not comparable to a full run.
• Errored rows are silent in the pass-rate denominator. If 30 rows out of 100 errored, and the remaining 70 all passed, the pass rate shows 100%. Always look at the Errors chip count.
• Read the judge's reasoning at least once per new metric. The fastest way to spot a rubric that isn't asking what you thought.
• CSV export gives you everything. When you need cross-cutting analysis the UI doesn't support, pull the CSV and use a spreadsheet or a notebook.
• The Compare view is on the test plan page, not on the run page. Tick two or more runs in the plan's Eval runs tab; the compare button appears in the toolbar.

Related

• Test plans hub: the plan defines what gets scored.
• Datasets hub: the rows the run iterates over.
• Metrics hub: the shape of each score.
• Eval runs page: UI reference for the workspace-wide list.