Agent metrics

The Agent category holds metrics designed for multi-turn conversations and tool-using workflows. Where the RAG category asks "was this one response any good?", the Agent category asks "did the whole interaction produce the right outcome?".

Most metrics here read the turns column (a full conversation transcript) instead of a single input / actual_output pair. A few straddle both (Task Success Rate, Tools Error). All except Tool Correctness use the plan's judge model.

Attach these to eval_multi_turn plans, or to single-turn plans that model a tool-using agent.

Tool Correctness

What it scores. Whether the agent called the right tools with the right arguments. No LLM judge, this is a structural comparison against an expected_tool_calls set on the row.

Columns required. tools_called.

Default threshold. 0.5.

When to use. Any agent whose success is defined by external actions: booking, ordering, searching a database, sending an email. If the agent called the wrong tool or passed wrong args, the "answer" was already wrong regardless of what the response text said.

When not to use. Chat-only products with no tool use.

Parameters.

• evaluation_params: subset of argument keys to compare. Empty = compare every argument. Useful when the agent legitimately fills some args by inference (a timestamp, a request id) and you only want to check the intent-bearing ones.
• exact_match: require every tool call to match exactly. Off by default, partial matches earn partial credit.
• check_ordering: require tool calls to appear in the same order as expected. Off by default, use for workflows where the order of operations matters (email confirmation after payment, not before).

Tip. Combine with Tools Error (below): Tool Correctness catches "wrong tool", Tools Error catches "right tool, called badly".

Task Success Rate

What it scores. Whether the agent actually accomplished the user's task end-to-end. LLM-judged over the full transcript.

Columns required. input, actual_output, turns (works on both single-turn and multi-turn plans).

Default threshold. 0.7.

When to use. The primary "did the product work?" metric for multi-turn agents. Simple to reason about, easy to communicate to non-technical stakeholders.

When not to use. Skip on plans where "success" is subjective (open-ended conversation, tutoring). Reach for G-Eval or Custom Eval with your own rubric instead.

Parameters.

• task_description: optional. When set, skips the goal-inference LLM call and uses your description directly. Cheaper and more reproducible.
• success_criteria: optional list of explicit success criteria. When set, skips the criteria-generation LLM call. Recommended for reproducibility.

Goal Achievement

What it scores. Whether the user actually got what they wanted from the conversation. Outcome-oriented sibling of Task Success Rate: task completion is about the agent's checklist; goal achievement is about the user's satisfaction.

Columns required. turns.

Default threshold. 0.7.

When to use. Support and sales conversations where the user came in with a goal (refund an order, book a demo, find an answer). Task Success Rate scores whether the agent did its job; Goal Achievement scores whether the user left with what they came for. The two often disagree in interesting ways: an agent can technically complete the flow while the user still leaves confused.

When not to use. Tasks with no clear user goal (open conversation, exploratory research).

Parameters.

• user_goal: optional. When set, skips the goal-inference LLM call. Recommended for reproducibility, same idea as task_description on Task Success Rate.

Role Adherence

What it scores. How consistently the agent stays in its assigned persona across the conversation. Breaking character mid-conversation scores down.

Columns required. turns.

Default threshold. 0.7. Consider raising to 0.95 for brand-sensitive products, character breaks in customer-facing agents are a serious bug.

When to use. Any agent with an assigned persona: support agent, sales rep, tutor, health assistant. Especially important when the persona is a brand promise (a friendly-but-professional support voice).

When not to use. Neutral chat products with no persona.

Parameters.

• chatbot_role: text description of the persona the chatbot must adhere to. Falls back to the row's chatbot_role column when empty. Setting it here is recommended for consistency across rows.

Knowledge Retention

What it scores. Whether the agent remembers facts stated earlier in the conversation. Detects contradictions and lapses across turns.

Columns required. turns.

Default threshold. 0.7.

When to use. Multi-turn agents where the user provides info incrementally (an order number in turn 1, a product name in turn 3, a delivery date in turn 5). This is the metric that catches context-loss bugs.

When not to use. Single-turn plans (no memory to check). Also skip on short conversations (2-3 turns), the metric needs some length to have signal.

Tools Error

What it scores. Errors in tool use across the conversation: parameter_error, invalid_function, sequence_error, result_ignored, repeated_failure, error_handling. Complements Tool Correctness (which scores whether the right tools were called).

Columns required. input, actual_output, expected_output.

Default threshold. 0.7.

When to use. Any tool-using agent. Tool Correctness catches "wrong tool"; Tools Error catches "right tool, but the agent screwed up how it used it".

When not to use. Non-tool-using agents.

Parameters.

• error_types: subset of the six categories above to flag. Empty = check every category.

Conversational Flow

What it scores. Overall dialogue naturalness and coherence. Penalises unnecessary clarifications, repetition, and stilted phrasing.

Columns required. turns.

Default threshold. 0.7.

When to use. Consumer-facing chat and voice products where the experience of the conversation matters, not just the outcome. A support bot that answers every question correctly but takes 8 turns to do so scores badly here.

When not to use. Low-touch flows (a two-turn API-style agent that takes a request and returns a result). Conversational quality isn't the right question.

Failure Rate

What it scores. How the agent handles uncertainty. Does it hallucinate a confident answer, stall out, or honestly propose alternatives? Higher = better handling.

Columns required. turns.

Default threshold. 0.7.

When to use. High-stakes agents where confidently-wrong answers are worse than "I don't know" (medical, legal, financial, technical support). Pairs well with Faithfulness (which catches groundless claims per response) to give a full picture of uncertainty handling.

When not to use. Casual conversation products where a plausible guess is fine.

Goal Achievement Rate

Alias for Goal Achievement, described above.

Repetitive Pattern Detection

What it scores. Loops where the agent repeats the same actions or responses without progressing. High score = no loops; low score = the agent is stuck.

Columns required. turns.

Default threshold. 0.7.

When to use. Multi-turn agents, especially tool-using ones (a loop of "call tool, fail, call tool, fail" is a common bug). Add it to any plan where conversation length is unbounded.

When not to use. Fixed-length conversations (say a 3-turn scripted plan), no room for loops to manifest.

How to pick a starting set

For a multi-turn evaluation plan:

• Support / help-desk agent: Task Success Rate + Goal Achievement + Conversational Flow. Covers "did the job get done", "was the user happy", "was the experience OK".
• Tool-using agent (booking, purchasing, data lookup): Tool Correctness + Tools Error + Task Success Rate. Structural + LLM-judged checks together.
• Persona-driven agent (branded support voice, tutor): add Role Adherence on top of the above.
• Long-conversation agent (10+ turns typical): add Knowledge Retention and Repetitive Pattern Detection.
• High-stakes agent (medical, legal, finance): add Failure Rate and a Custom Eval rubric for your domain compliance rules.

Three metrics is a healthy starting count. Every LLM-based metric is one extra judge call per conversation, and multi-turn plans already have the target's own N-turn cost. Restraint pays off in dollars.

Tips and pitfalls

• Multi-turn metrics need turns, not input / actual_output. If you're seeing "column missing" errors on the Agent category, the dataset was probably generated for single-turn plans.
• Task Success Rate can silently succeed for the wrong reason. If the judge infers a lax task_description, an agent that partially completes the task can still pass. Set task_description and success_criteria explicitly for reproducibility.
• Role Adherence at 100% across everything usually means the chatbot_role description was too loose to fail against. Tighten it.
• Knowledge Retention needs multi-fact conversations. If your dataset's conversations are single-fact (user asks one thing, gets one answer), the metric has nothing to detect and scores trivially high.
• Combine Tool Correctness with Tools Error. Neither alone catches every tool-use bug: Correctness misses argument-shape errors, Tools Error misses "called the wrong tool entirely".
• Repetitive Pattern Detection is noisy on short conversations. Skip it on plans where max_turns is 3 or less.

Related

• Metrics hub
• RAG and general LLM-judge metrics
• Custom metrics
• Evaluation, multi-turn
• Personas and escalation styles