How we evaluate AI systems, and what the metrics actually measure.
6 posts in Expertise tagged Custom metric. Clear filters
A metric threshold is a starting point, not a fact about your AI system. This guide shows how to label a calibration set, place the threshold, check the LLM judge rubric and separate noise from signal.
Read the postA RAG AI system that scores well on happy-path rows can still answer a question it should decline, answer the neighbouring question, or quote a superseded policy. This guide shows how to build the three trap rows that catch each failure and how to score them in EvaliQA.
Read the postA custom LLM judge for an AI system can be written as one holistic rubric or as a list of discrete criteria, and the shape decides what the AI evaluation score can tell you. We explain how EvaliQA scores each, what each hides, and how to pick the shape for a given metric.
Read the postAn AI judge criterion that never references the row it scores can only judge plausibility, and in EvaliQA Custom Eval it is not judged at all. Placeholders make one AI metric read a different, concrete sentence on every row, and this post covers where the values come from and how to check them.
Read the postPrompt injection sits at the top of the OWASP Top 10 for LLM applications, and most teams test only the direct form. Here is how to test direct attacks, payloads hidden in retrieved data and multi-turn build-ups in EvaliQA, and what breach rate to accept before you ship.
Read the postPosition, verbosity and self-preference bias make an LLM judge grade the wrong things. Here is how each one shows up in a run and how to build a Custom Eval metric in EvaliQA whose criteria leave the judge no room to act on them.
Read the post
