How we evaluate AI systems, and what the metrics actually measure.
20 posts in Expertise tagged LLM judges. Clear filters
Deterministic rules, semantic similarity and LLM judges each answer a different question about an AI system. Here is how to stack them so the cheap checks carry the weight and the judge does only what nothing else can.
Read the postA metric threshold is a starting point, not a fact about your AI system. This guide shows how to label a calibration set, place the threshold, check the LLM judge rubric and separate noise from signal.
Read the postSingle-output scoring and pairwise comparison answer different questions about an AI system, and most teams pick one by habit. Here is how to choose between an absolute score and a preference, what each LLM judge pattern misses, and how to run each without fooling yourself.
Read the postA chunking or embedding change rewrites what your AI RAG system retrieves for every query at once, and spot checks will not show you what broke. This is the regression protocol we use to compare the old and new pipeline query by query before the change ships.
Read the postA CI gate for AI agent regression testing is only useful if it fails on real regressions and stays green when the LLM judge wobbles. We measure the noise floor with two identical runs, pick metrics that flip for real reasons and set the minimum pass rate with room for noise.
Read the postA custom LLM judge for an AI system can be written as one holistic rubric or as a list of discrete criteria, and the shape decides what the AI evaluation score can tell you. We explain how EvaliQA scores each, what each hides, and how to pick the shape for a given metric.
Read the postAn AI judge criterion that never references the row it scores can only judge plausibility, and in EvaliQA Custom Eval it is not judged at all. Placeholders make one AI metric read a different, concrete sentence on every row, and this post covers where the values come from and how to check them.
Read the postA coding agent that edits a test to make it pass still produces an ordinary pull request. This post shows how to trace every Claude Code or Codex session into EvaliQA, score it as it arrives with detectors, metrics and the AI Analyst, and promote the bad sessions into your test dataset.
Read the postWhen a RAG answer is wrong, Faithfulness alone cannot tell you whether retrieval or generation failed. Context recall and context precision score the retrieved chunks themselves, so you fix the stage that actually broke.
Read the postAnswer Relevancy and Faithfulness get read as one quality score, but they read different columns and catch different bugs. This post shows what each one sees, what it misses, and how to read the two scores together when they split.
Read the postPosition, verbosity and self-preference bias make an LLM judge grade the wrong things. Here is how each one shows up in a run and how to build a Custom Eval metric in EvaliQA whose criteria leave the judge no room to act on them.
Read the postA RAG answer can fail in the retriever or in the generator, and an end-to-end score cannot tell you which. Here is how the retrieval and generation metrics in EvaliQA split the two stages, and how to read them in pairs.
Read the post
