How we evaluate AI systems, and what the metrics actually measure.
20 posts in Expertise tagged Method. Clear filters
Deterministic rules, semantic similarity and LLM judges each answer a different question about an AI system. Here is how to stack them so the cheap checks carry the weight and the judge does only what nothing else can.
Read the postA metric threshold is a starting point, not a fact about your AI system. This guide shows how to label a calibration set, place the threshold, check the LLM judge rubric and separate noise from signal.
Read the postSingle-output scoring and pairwise comparison answer different questions about an AI system, and most teams pick one by habit. Here is how to choose between an absolute score and a preference, what each LLM judge pattern misses, and how to run each without fooling yourself.
Read the postA chunking or embedding change rewrites what your AI RAG system retrieves for every query at once, and spot checks will not show you what broke. This is the regression protocol we use to compare the old and new pipeline query by query before the change ships.
Read the postA CI gate for AI agent regression testing is only useful if it fails on real regressions and stays green when the LLM judge wobbles. We measure the noise floor with two identical runs, pick metrics that flip for real reasons and set the minimum pass rate with room for noise.
Read the postA RAG AI system that scores well on happy-path rows can still answer a question it should decline, answer the neighbouring question, or quote a superseded policy. This guide shows how to build the three trap rows that catch each failure and how to score them in EvaliQA.
Read the postA custom LLM judge for an AI system can be written as one holistic rubric or as a list of discrete criteria, and the shape decides what the AI evaluation score can tell you. We explain how EvaliQA scores each, what each hides, and how to pick the shape for a given metric.
Read the postA production AI failure you fixed but never wrote an AI test for comes back with the next prompt change or AI update. This is how to turn the trace behind it into a golden-set row, with an expected output and a metric that catch the failureof AI, that CI checks on every pull request.
Read the postAn AI chatbot that passes your own tests has only been tested by one kind of user. Persona-driven simulation plays 15 defined user types against it for whole AI conversations, and the gap between their pass rates is where the bugs are.
Read the postA spreadsheet of public jailbreak prompts tells you whether your AI system refuses those strings, not whether an attacker can get what they came for. How to red-team from attack objectives instead, and how EvaliQA turns them into a breach rate per vulnerability and technique.
Read the postA scheduling AI agent can say "Booked, Tuesday at 10:00" when the booking tool was never called, returned an error or booked the wrong day, and AI judge that reads only the reply passes all three. This is a 7-step procedure for scoring the tool calls, arguments, results and handling instead.
Read the postPosition, verbosity and self-preference bias make an LLM judge grade the wrong things. Here is how each one shows up in a run and how to build a Custom Eval metric in EvaliQA whose criteria leave the judge no room to act on them.
Read the post
