How we evaluate AI systems, and what the metrics actually measure.
24 posts in Expertise tagged Metrics. Clear filters
Deterministic rules, semantic similarity and LLM judges each answer a different question about an AI system. Here is how to stack them so the cheap checks carry the weight and the judge does only what nothing else can.
Read the postA metric threshold is a starting point, not a fact about your AI system. This guide shows how to label a calibration set, place the threshold, check the LLM judge rubric and separate noise from signal.
Read the postSingle-output scoring and pairwise comparison answer different questions about an AI system, and most teams pick one by habit. Here is how to choose between an absolute score and a preference, what each LLM judge pattern misses, and how to run each without fooling yourself.
Read the postA chunking or embedding change rewrites what your AI RAG system retrieves for every query at once, and spot checks will not show you what broke. This is the regression protocol we use to compare the old and new pipeline query by query before the change ships.
Read the postA CI gate for AI agent regression testing is only useful if it fails on real regressions and stays green when the LLM judge wobbles. We measure the noise floor with two identical runs, pick metrics that flip for real reasons and set the minimum pass rate with room for noise.
Read the postAn AI agent's reply can read well while the tool calls behind it were wrong. This guide shows how to check the tool, the arguments and the order with a structural metric that needs no LLM judge, and where that check stops.
Read the postA RAG AI system that scores well on happy-path rows can still answer a question it should decline, answer the neighbouring question, or quote a superseded policy. This guide shows how to build the three trap rows that catch each failure and how to score them in EvaliQA.
Read the postA custom LLM judge for an AI system can be written as one holistic rubric or as a list of discrete criteria, and the shape decides what the AI evaluation score can tell you. We explain how EvaliQA scores each, what each hides, and how to pick the shape for a given metric.
Read the postA production AI failure you fixed but never wrote an AI test for comes back with the next prompt change or AI update. This is how to turn the trace behind it into a golden-set row, with an expected output and a metric that catch the failureof AI, that CI checks on every pull request.
Read the postAn AI chatbot that passes your own tests has only been tested by one kind of user. Persona-driven simulation plays 15 defined user types against it for whole AI conversations, and the gap between their pass rates is where the bugs are.
Read the postAn AI judge criterion that never references the row it scores can only judge plausibility, and in EvaliQA Custom Eval it is not judged at all. Placeholders make one AI metric read a different, concrete sentence on every row, and this post covers where the values come from and how to check them.
Read the postA scheduling AI agent can say "Booked, Tuesday at 10:00" when the booking tool was never called, returned an error or booked the wrong day, and AI judge that reads only the reply passes all three. This is a 7-step procedure for scoring the tool calls, arguments, results and handling instead.
Read the post
