Finding bottlenecks

Aggregate pass rate tells you whether something is wrong. Bottleneck analysis tells you where. A run at 65% pass rate is not one problem, it's a distribution of problems, and the useful question is always "which slice of the run is dragging the average down?".

This page is the methodology of that slicing. Four common bottleneck dimensions, how to spot each on the run detail page, and what each pattern usually means.

The four dimensions to slice by

Every failing row belongs to several slices at once: a metric, a dataset category, a persona (multi-turn), a mode (single-turn or scripted). Bottlenecks live at the intersections; find them by looking one dimension at a time.

1. By metric

Where to look. The per-metric charts under the KPI tiles. Each metric has a score distribution with a threshold marker. Metrics with a lot of mass below the marker are the ones dragging pass rate down.

Common patterns.

• One metric is 30+ pp below the others. That one metric captures the current bottleneck. Read the judge's reasoning on 5-10 failing rows for that metric; the failure story usually clusters.
• Every metric hovers around 60-70%. Not a single-metric bug; either the target is broadly weak on this dataset, or the thresholds are too tight for the current model / prompt. Try loosening one threshold to see which numbers move.
• Faithfulness low, Answer Precision fine. The model is answering right but with ungrounded claims. Common in RAG products where the target has enough general knowledge to guess correctly even when the retrieved context is wrong. Bad on trust; fix retrieval.
• Answer Precision low, Faithfulness fine. The model is grounded in the context but drawing the wrong conclusion from it. Prompt or model issue, not retrieval.
• Role Adherence low, Task Success Rate fine. The agent is completing tasks but breaking character while doing it. Prompt tone / persona issue.
• Tool Correctness low, Tools Error fine. The agent is calling wrong tools (but calling them cleanly). Usually a system prompt problem where tool descriptions overlap or are ambiguous.
• PII Leakage or Toxicity fails on any row. Not a fuzziness metric, treat every failure as an incident. Read the row, patch the product, add a row to the golden set.

2. By category / tag

Where to look. Row expand view, look at the row's category or tag column (whatever metadata you added). To slice by category, use the CSV export and pivot in a spreadsheet; the UI doesn't group by tag natively yet.

Common patterns.

• One category at 40% pass, others at 90%. That category is the bottleneck. Zero in on it: what makes those rows different from the passing ones? Same input length? Same intent? Same required tool?
• Categories fail proportionally to their difficulty. Easy rows pass, hard rows fail. Working as intended, but worth checking that the difficulty labels are accurate.
• A "trivial" category is failing. Something regressed. Trivial rows should always pass; a failure there is often a signal of a broken prompt or a wrong-context injection.

How to structure categories for this to work. Assign a tag to every row you write or promote from production. Two useful axes: intent (billing.refund, support.escalation) and difficulty (easy, medium, hard). Even a rough labelling gives you slicing that the aggregate hides.

3. By persona (multi-turn only)

Where to look. For multi-turn plans, every row has an implicit persona (the one the generator used). Read the transcript in the row expand to see the persona's tone and pace.

Common patterns.

• Default persona passes; Impatient persona fails. The agent needs many turns to satisfy a request; impatient users abandon. Fix by making the agent answer earlier, ask fewer clarifying questions, or handle terse inputs better.
• Non-native speaker persona fails much more than Default. Intent classification is brittle to phrasing. Worth checking upstream NLU or providing more paraphrasing in the prompt.
• Confused persona fails on Knowledge Retention. The agent is losing context when the user contradicts themselves. Common in agents with short memory windows or aggressive summarisation.
• Aggressive persona fails on Role Adherence. The agent mirrors the user's tone. If your product must stay polite regardless of input, this is a prompt tuning target.

4. By turn count / mode

Where to look. For multi-turn plans, look at the "average turns per conversation" (visible in row details). For single-turn plans, look at avg_input_tokens and avg_output_tokens distributions in row expand.

Common patterns.

• Multi-turn: avg turns much higher than expected. A 3-turn scenario using 7 turns to resolve means the agent isn't closing. Cost + latency + Conversational Flow all suffer.
• Multi-turn: pass rate drops at high turn counts. Long conversations lose context; Knowledge Retention is the metric to add.
• Single-turn: pass rate drops on long inputs. The model is losing important detail mid-input. Try chunking, or move to a model with better long-context handling.
• Single-turn: pass rate drops on short inputs. The model needs more context than the user provided. The agent probably should ask clarifying questions, or the intent classifier upstream needs more training data.

The three non-quality bottlenecks

Bottleneck analysis isn't just about pass rate. Cost, latency, and coverage each have their own patterns.

Cost bottleneck

Signal. Total cost KPI is significantly higher than the previous run or the budget.

Where to look.

• Per-row token usage in the row expand view. Which rows are the 10% most expensive? Are they meaningfully harder, or is the model just generating longer answers?
• Metric configuration. Every LLM-based metric is one judge call per row. G-Eval defaults to 20 samples per row (20 judge calls per row). Custom Eval consensus multiplies by n_runs. Cost creep often traces to a metric added without checking the math.
• Prompt or context bloat. If avg_input_tokens grew from 500 to 2500 between runs, someone added a long system prompt or the retrieval pipeline is returning more chunks than before.

Fixes. Trim retrieved chunks, drop redundant metrics, reduce n_samples on G-Eval, switch to a cheaper judge for iteration and back to the expensive one only for release runs.

Latency bottleneck

Signal. Avg latency KPI is significantly higher than the previous run or your SLA.

Where to look.

• Per-row Response time in the row expand. Which rows are the 10% slowest? Is there a pattern (long input, specific tool call, specific category)?
• Sequence of tool calls for agent modes. Are agents making serial calls where parallel would work? Are they retrying failed calls needlessly?
• Model change. Some flagship models are slower than others; a swap can trade off cost, quality, and latency in surprising ways.

Fixes. Cache retrieval results, parallelise independent tool calls, cap context size, or accept the slower model if quality matters more than latency.

Coverage bottleneck

Signal. Your dataset is small or one-dimensional. Pass rate is high but the run isn't testing the important cases.

Where to look.

• Dataset size vs your product's surface area. If your product has 20 business scenarios and your dataset has 30 rows, most scenarios have one row each. A single-row scenario tells you almost nothing.
• Row distribution across categories. If 80% of rows are one category, the aggregate pass rate is really the pass rate on that category.
• Row age. If most rows are 6 months old, they may not reflect the current scope of the product.

Fixes. See Datasets, methodology hub for the cadence of growing coverage.

The "rerun before you fix" rule

Before you form a diagnosis from one run, especially one that looks weirdly bad, run the plan again unchanged. Two possible outcomes:

• The numbers are stable across runs. The bottleneck is real. Start iterating on the product.
• The numbers moved significantly between identical runs. The bottleneck is judge variance, not a product bug. Reduce it by lowering judge temperature, adding consensus (Custom Eval n_runs), or using a stronger judge model.

Two runs is cheap; chasing an imaginary bottleneck for a week is not.

Common analysis mistakes

• Reading the aggregate as if it were a diagnosis. 78% pass is not "the product is 78% good". It's "somewhere in this run there are failures, go look at them".
• Fixing the first failure you see. Individual failures are rarely the story. Read 5-10 in a category before deciding what the pattern is.
• Ignoring the judge's reasoning. The judge tells you why it scored the way it did. Read it. Half the time the "bug" is the metric asking the wrong question.
• Only looking at Failed. Rows that passed but only barely (score at the threshold + 0.01) are next month's regressions. Sort by score ascending and read the borderline ones.
• Chasing a percentage-point drop that's within judge variance. A 2 pp drop between two runs on the same plan often is noise, not regression. Rerun the previous plan; if the number moves, it was noise.
• Blaming the model when the dataset changed. Dataset drift can masquerade as model regression. Freeze the dataset version between runs when you're comparing.

Related

• Evaluation results: the hub.
• Comparing runs: the next step after finding the bottleneck.
• Preparing a report: how to communicate the bottleneck once you've found it.
• Metrics hub: which metric is the right question to ask.
• Personas and escalation styles: slicing by persona in multi-turn plans.