How we evaluate AI systems, and what the metrics actually measure.
8 posts in Expertise tagged Agents. Clear filters
An AI agent's reply can read well while the tool calls behind it were wrong. This guide shows how to check the tool, the arguments and the order with a structural metric that needs no LLM judge, and where that check stops.
Read the postA CI gate for AI agent regression testing is only useful if it fails on real regressions and stays green when the LLM judge wobbles. We measure the noise floor with two identical runs, pick metrics that flip for real reasons and set the minimum pass rate with room for noise.
Read the postA production AI failure you fixed but never wrote an AI test for comes back with the next prompt change or AI update. This is how to turn the trace behind it into a golden-set row, with an expected output and a metric that catch the failureof AI, that CI checks on every pull request.
Read the postAn AI chatbot that passes your own tests has only been tested by one kind of user. Persona-driven simulation plays 15 defined user types against it for whole AI conversations, and the gap between their pass rates is where the bugs are.
Read the postA spreadsheet of public jailbreak prompts tells you whether your AI system refuses those strings, not whether an attacker can get what they came for. How to red-team from attack objectives instead, and how EvaliQA turns them into a breach rate per vulnerability and technique.
Read the postA coding agent that edits a test to make it pass still produces an ordinary pull request. This post shows how to trace every Claude Code or Codex session into EvaliQA, score it as it arrives with detectors, metrics and the AI Analyst, and promote the bad sessions into your test dataset.
Read the postA scheduling AI agent can say "Booked, Tuesday at 10:00" when the booking tool was never called, returned an error or booked the wrong day, and AI judge that reads only the reply passes all three. This is a 7-step procedure for scoring the tool calls, arguments, results and handling instead.
Read the postAgents fail across turns in ways no single-turn check can see: they repeat a tool call, lose what the user said, and drift out of their role under pressure. This guide shows how to catch all three in EvaliQA, offline with multi-turn scenarios and live with online evaluation.
Read the post
