Score the AI agent's path, not its final message

A patient writes to a clinic's scheduling agent, "I need to see Dr Novak next week, Tuesday morning if you can." Two tool calls later the agent answers, "Booked. Tuesday 20 October at 10:00 with Dr Novak. You will get a reminder the day before." The reply is specific, polite and exactly what a working agent would say, and in three different runs it is untrue. In the first run the agent looked up the free slots and never called the booking tool. In the second it called the booking tool, the tool answered that the slot had just been taken, and the agent confirmed anyway. In the third it booked 27 October instead of 20 October. The patient turns up on the 20th, the receptionist finds nothing, and the evaluation run that signed off this version of the agent reported a pass on every one of those rows.

The pass is not a bug in the judge. It is what happens when evaluation reads two fields, the user's input and the agent's final output, and the output is identical in a correct run and in all three failures. The evidence lives in what the agent did between the request and the reply, which tools it called, with what arguments, what came back and what it did next. We call that the path. This post is a procedure for scoring it in EvaliQA, in 7 steps, from writing down what a correct run looks like, through the three metrics that each read a different layer of the path, Tool Correctness, Tools Error and Task Success Rate, to watching the same path on production traffic in online evaluation. We use the booking agent throughout, bring in an IT helpdesk agent and a delivery agent where the shape of the failure differs, and close with what the method does not catch.

Where the evidence actually lives

The model that writes the closing message is the same model that decided whether to call book_slot, and it writes the message by predicting what a good closing looks like after a conversation of this shape. Nothing in that step checks that the tool ran or that it returned what the message claims. When the tool returns an error, the error text sits in the context window as one more input, and a model that has produced thousands of confirmations will often produce another one. When the call was skipped there is not even an error to ignore. The closing message is a template that fires on the shape of the conversation, not on the state of the calendar.

The path has four layers and the three failures each break a different one. Tool selection is which tools were called and in what order. Arguments are what was passed to each. Results are what each tool returned, including errors. Handling is what the agent did after each result, whether it retried, changed course, told the user or carried on. A skipped call breaks selection, since book_slot is missing. An ignored error breaks handling, since the result said "slot taken" and the next step was a confirmation. A wrong argument breaks the second layer, since the call went out with 27 October. In an EvaliQA test run the connector reports the tools it called alongside the answer, and in production the trace page shows the same thing as a span tree, where a span is one timed step such as a model call or a tool call, with a collapsible "Tools called" list.

Scoring the message cannot separate these runs because the message carries no information about any of the four layers. Answer Relevancy passes all three, since the reply is squarely on topic. Faithfulness does not apply, since there is no retrieval context to be faithful to. A human reviewer who reads only the last turn passes them too, which is how agents like this reach production. A pass rate measured on the final message is therefore an upper bound. It measures how convincingly the agent closes, and the gap between that number and the rate at which the work got done is exactly the set of complaints that reach the clinic's front desk. Scoring the path costs one thing the message does not, a reference for what the path should have been, and that is where the procedure starts.

1. Write down what a correct run does

Pick the scenarios where success is an action in the world rather than an answer in the chat. Appointment booked, password reset, delivery address changed, ticket escalated. These are the scenarios where the message can diverge from reality, and they are the only ones that need a path reference. For each one, write the tools a correct agent calls, with the arguments that carry the user's intent, in order if the order is part of the contract. For the booking the correct path is find_slots with provider Novak and the date range the patient asked for, then book_slot with provider Novak, a start of 20 October at 10:00 and the patient id, then send_sms to the patient. Leave out arguments the agent fills in by itself, such as an idempotency key or a request timestamp, because they differ on every run and a comparison that includes them fails correct behaviour.

The other two agents show how the reference changes shape. The helpdesk agent gets "I am locked out of my account, can you reset my password?". The correct path is lookup_account, then reset_password, and a temporary password sent to the recovery address. But if the account is locked for a security reason, reset_password returns a refusal, and the correct next step is open_ticket and an honest message with the ticket number, not "your password has been reset". The delivery agent gets "change the address on my order to 14 Mill Lane", and the correct path is update_address with the order id the user gave earlier in the conversation and the new address. The argument that matters is the order id, since calling the tool on the wrong order updates a stranger's parcel while confirming yours.

Next to the tools, write the expected output as the state the world should be in and the message that state supports. For the booking row that is "an appointment for patient 8841 on 20 October at 10:00 with a booking id, confirmed to the patient with that date and time". Then add failure rows, which are the ones that catch the ignored error directly. If your connector points at a sandbox where book_slot can be made to return "slot taken" for a given start time, write a row for that slot whose expected output is an honest message and the next free slot, and whose success criteria forbid any claim that the appointment exists. The same goes for the locked account on the helpdesk agent. Tag every row, for instance scheduling.book and scheduling.book.failure, so you can slice the run by scenario later.

Five rows per scenario is enough to start, two clean passes, two rows where the tool fails and one where the request is ambiguous, say a patient who asks for "next week" without a day. The dataset will grow from production once the loop in step 7 is running, and rows promoted from real traffic are better than anything you can imagine at a desk. What matters now is that every row carries a path reference the metrics below can read, because without it the three metrics either have nothing to compare against or have to infer the task themselves, and inference is what let the false confirmation through in the first place.

2. Make the path reach the judge

Every path metric reads only what arrives with the row. In a test run the connector calls your AI system, receives the answer, and reports the answer together with the tools the system called. Tool Correctness reads that called-tools list. Tools Error reads the agent's steps. If your endpoint returns only the final text, both metrics are as blind as a message-only judge, and a plan that attaches them will either score every row the same or fail with a missing column. Before writing a single criterion, check what the connector actually returns for one booking request in the playground and confirm that the tool calls, their arguments and their results are in the response.

The results matter as much as the calls. A record that lists book_slot without what it returned lets Tool Correctness do its job but leaves Tools Error unable to see that the return was "slot taken". Make the tool result travel with the trace, either inside the step record or as a separate field. For the outcome, add an output field to the connector, for example booking_status mapped from the booking status in your API's response in the connector's Response tab. That field becomes available as a placeholder, {{booking_status}}, in Custom Eval criteria on every run that uses the connector, with nothing to configure on the plan. It is the only way a judge can check an outcome rather than infer one, and step 5 relies on it.

In production the same requirement holds, with spans instead of a connector response. The SDK or an OpenTelemetry exporter sends a trace per request, and a tool span that records its input, output and an error status is what the detectors and the trace page work from. A tool span that records only its name tells you the agent called book_slot and nothing about whether the booking exists. Wire the result and the status into the span now, because online evaluation does not re-score history, and the first real case of the agent claiming a booking that does not exist is the one you will most want to read in full.

The last piece is the column mapping on the run sheet. When you start a run you map the input column, the expected output column and, for tool metrics, the expected tools. A dataset column literally named expected_tools is mapped without asking. Any other name you pick in the field called "Expected tools column (optional)". The values can be a list or text such as "find_slots, book_slot, send_sms". If the plan has a Custom Eval metric that references a dataset column as a placeholder, tick that column under "Custom-metric extra fields (optional)", which appears on the run sheet only when the plan has a Custom Eval or G-Eval metric.

3. Compare the calls against the reference

Tool Correctness reads the first two layers of the path. It uses no LLM judge. It compares the tools the connector reported, with their arguments, against the expected tools on the row and returns a score from 0 to 1, with a default threshold of 0.5 and partial credit for partial matches unless you turn on exact_match. The evaluation_params parameter restricts the comparison to a subset of argument keys, which is how you compare the slot start and the patient id and ignore the idempotency key. The check_ordering parameter requires the calls in the expected order, which matters when the sequence is the contract, such as the confirmation text that must go out after the booking rather than before it. In the test plan wizard the metric is offered on single-turn plans, which for a tool-using agent means one user request, the agent's whole run of tool calls, and one final answer.

Because the comparison is structural, it is deterministic, cheap and identical from run to run, which makes it the metric for the plan that runs on every merge. It catches the skipped call outright, since book_slot is absent from the called list. It catches the wrong argument, since the slot start is 27 October and the reference says 20 October. On the delivery agent it catches update_address called with the wrong order id, which is the failure that touches another customer's parcel. All of these pass a message-only judge every time, and all of them are caught here with no judge cost and no variance. When this score drops after a prompt change, the change moved tool selection or argument construction, and the failing rows show you exactly which way.

What it does not catch is the ignored error. In that run the agent called the right tool with the right arguments, which is everything Tool Correctness looks at, and it never reads what the tool returned. A correctly placed call that failed scores the same as one that succeeded. It is also only as good as the reference. If the expected tools list was written for an older version of the tool set, say before find_slots was split into two tools, the metric fails correct behaviour and you will spend an afternoon chasing a regression that is in the dataset. Treat the expected tools column as code, review it whenever the tool set changes, and keep the compared arguments to the ones that carry intent.

4. Judge what the agent did with results

Tools Error reads the third and fourth layers, the results and the handling. It is LLM-judged, with a default threshold of 0.7, and it looks for six categories of mistake in the agent's steps, parameter_error, invalid_function, sequence_error, result_ignored, repeated_failure and error_handling. The error_types parameter restricts it to a subset, and an empty value checks all six. The row needs an input, the agent's output and an expected output, and in the wizard it sits with Tool Correctness on single-turn plans. The docs sum up the division of labour, Tool Correctness catches "wrong tool" and Tools Error catches "right tool, called badly", and neither alone covers every tool-use bug.

The ignored error is result_ignored and error_handling at once. The agent received "slot taken" from book_slot, carried on as if it had received a booking id, sent the SMS and told the patient the appointment exists. The helpdesk agent produces the same shape when reset_password refuses because the account is locked and the agent writes "your password has been reset". A related shape is repeated_failure, where the agent calls the failing tool 4 times in a row, gets the same refusal each time, and then declares success as a way out of the loop. And sequence_error is the agent that sends the reminder before it has booked anything. Tool Correctness passes every one of these, because the right tools were called with the right arguments. Tools Error is the metric that reads past the call to its result and asks whether the next step made sense.

Its limits follow from being a judge. It can only read the steps the connector reports, which is why step 2 insists on the results travelling with the trace. If the record carries tool names and no results, the judge cannot see that book_slot failed any more than a message-only judge could, and it will pass the run. It is also a judgement rather than a comparison, so the same row can score differently on two runs, and a run with a single tool call gives it little to work with. When you first attach it, read the judge's reasoning on a handful of failing rows and narrow error_types to the categories whose failures you would act on, because a finding you will never fix is a judge call you are paying for.

5. Write criteria the message cannot satisfy

Task Success Rate asks whether the agent accomplished the user's task end to end, judged over the full transcript, with a default threshold of 0.7, and it works on single-turn and multi-turn plans. Left to itself it has the weakness that matters most here. When task_description and success_criteria are empty, the judge first infers the task and what success would look like, and an inferred criterion such as "the agent confirmed an appointment" is satisfied by the false confirmation. The docs say the metric can silently succeed for the wrong reason, and the three booking failures are what that sentence means in practice.

The fix is to write the criteria yourself and to point each one at evidence the final message cannot fake. For the booking row we write three. The book_slot tool was called for 20 October at 10:00 and returned a booking id. The final message states the date and time the tool returned, not a date the agent assumed. If book_slot failed, the agent said so plainly and offered the next free slot instead of claiming an appointment. Now the judge has to find a booking id in the transcript to award the first criterion, so the skipped call and the ignored error both fail it, and the wrong argument fails the second. Setting the criteria also skips the inference call, which makes the metric cheaper and gives the same verdict for the same transcript across runs.

Goal Achievement, the outcome-oriented sibling, scores whether the user left with what they came for, and it is a multi-turn metric. It is useful for support conversations in general, and it is a poor sole metric for this failure. In the transcript the patient is told the appointment exists and says thank you, so from inside the conversation the goal looks achieved. The judge has no view of the calendar. Use Goal Achievement to catch the agent that completes the flow while the user leaves confused, which is a different problem, and lean on Task Success Rate with written criteria for the agent that reports success while failing.

You may want to write a Custom Eval criterion such as "the agent must have called the booking tool before confirming". Two facts about Custom Eval stop that. The judge never receives tools_called, even though the field exists on the test case, so a criterion about tool calls has nothing to read. And every criterion must contain at least one placeholder that resolves, otherwise it is silently dropped, and a metric whose criteria are all prose scores 0. What works is the connector output field from step 2. With booking_status exposed by the connector, "The response claims an appointment only if {{booking_status}} is confirmed" is a criterion the judge can check, because the evidence arrives with the row. The same pattern gives the helpdesk agent a reset_status field and the delivery agent an address_updated flag.

6. Run five rows, then read the pattern

Before running the full set, run the plan on the 5 rows from step 1 where you already know the verdict. The docs recommend this for custom metrics, and it is just as useful here, because the most common first-run problem is not the agent, it is the reference. A stale expected tools list fails every row. A forgotten column mapping scores Tool Correctness at 0 across the board. A success criterion phrased loosely passes the false confirmation you wrote the row to catch. All three show up on 5 rows in minutes, and none of them should be mistaken for a regression. If a result looks strange, rerun the plan unchanged before you form a diagnosis, since a number that moves between two identical runs is judge variance, not the agent.

With three metrics on the plan, the useful signal is rarely one score. Each metric reads a different layer of the path, so the combination points at the layer that broke. The bottleneck guide lists some of these patterns, and the booking agent gives them concrete meaning.

The first row of the table is the one this post is about, and it is the one a message-only plan can never show you. Tool Correctness passing while Tools Error and Task Success Rate fail is the signature of an agent that did the right thing and then misreported the result, and it points at the handling layer rather than at selection or at the prompt's tone. The second row is the opposite trap. If Task Success Rate passes while Tool Correctness fails, the first question is not why the agent skipped the call, it is why the success criteria did not notice. Fix the criteria, rerun, and the second row usually turns into the third. In every case the next step is the same, open 5 to 10 failing rows for the metric that moved, read the judge's reasoning where there is one and the called tools where there is not, and resist loosening a threshold to make a pattern go away. A threshold moves the pass rate, it does not change which layer of the path is broken.

7. Watch the same path in production

The offline plan tests the scenarios you wrote. The agent that claims a booking it never made also does it on scenarios you did not write, so the same reasoning has to run on real traffic. Online evaluation scores every session that arrives with a session_id. On the project's Settings tab, the "Runtime auto-eval" panel has three switches, "Auto-evaluate sessions", "Deterministic detectors" and "AI Analyst". The detectors cost nothing and need no model. The error finding fires on any span with an error status, which is the failed book_slot call, and it escalates from warn to critical at 3 occurrences. The retry_loop finding fires on runs of identical adjacent spans with the same name and input, which is the agent hammering a refused reset_password. The redundancy finding fires when the same tool is invoked with the same arguments more than once in a session. The session-level pass runs 60 seconds after the last trace, so a conversation is analysed once, after it ends.

Metric bindings under "Metric scoring" behave differently from the offline plan, for a reason worth understanding. Tools Error can be bound to live traces, because it judges the agent's own steps and a real trace is the best source of those. Tool Correctness is not offered for a new binding, because a production trace carries no expected tools and the binding would compare every trace against an empty list. Task Success Rate and the other conversation-level metrics are bound as conversation metrics, where the picker asks "What one turn is made of" and EvaliQA assembles the dialogue from the session, and the rule about written criteria applies just as it does offline. Use the rule scope to limit judging to traces that touch the booking path, for instance a condition on a metadata field your app sets when a booking is attempted, rather than scoring every "what are your opening hours" at judge prices.

When a finding or a score points at a session, the pages under Runtime eval give you the path directly. The session page has the Analysis panel with the findings and conversation metrics, a timeline of every trace in wall-clock order, and the list of traces with an error badge on any that failed. The trace page has the span tree with an error banner on the span that failed, the "Tools called" list, a "Trace metrics" card with each metric's score, threshold and the judge's reasoning, and the findings that concern this trace. This is where you confirm which of the three runs you are looking at. It is also where "Add trace to dataset" turns the case into a row in the offline plan. You pick the test plan, the dataset and its version, map the trace fields to columns, and type the expected output you now know the agent should have produced as a custom value. The next run covers it.

Two limits to keep in mind. Online scoring is detection, not a controlled experiment. Traffic shifts, so a moving pass rate can reflect your patients as much as your agent, and the offline plan is still where you measure a change. And changing a binding affects traces from then on, history is not re-scored, so a detector you switch on today tells you nothing about last week. The loop that works is to let the detectors and Tools Error flag sessions, read the path on the trace page, promote the trace, and let the offline plan with Tool Correctness and written success criteria hold the line from there.

What this approach does not catch

The path is only as truthful as the tools that wrote it. If book_slot returns a booking id and the backend then fails to commit the appointment, every layer of the path looks correct, all three metrics pass, and the patient still finds nothing at the desk. No evaluation metric can see a database it is not shown. The only defence is to read the state back. A connector output field that queries the calendar after the run and reports whether the appointment exists, exposed as {{booking_exists}} to a Custom Eval criterion, turns the question from "did the tool say yes" into "is the appointment there", and the same read-back works in production as a final span. This is extra work on your side of the connector, and for the actions that matter most it is worth it.

The reference can also be too specific. A correct agent might call find_slots twice when the first range is empty, or ask a clarifying question before booking, and a reference that insists on exactly three calls in exactly one order fails it. Turning on exact_match and check_ordering makes this worse, so use them only where the order is genuinely the contract and let evaluation_params narrow the comparison to the arguments that carry intent. When Tool Correctness fails a row and the transcript reads fine, the row is telling you about the reference, and the fix is in the dataset.

And two of the three metrics are judges. Tools Error and Task Success Rate can score the same transcript differently on two runs, which is why the five-row check and the rerun-before-you-fix rule are part of the procedure rather than a nicety. Written success criteria and a narrowed error_types reduce the variance, they do not remove it. The structural check is the one you can gate a merge on without a second thought. The judged checks are the ones that tell you why, and reading their reasoning is part of using them.