Announcements, hands-on expertise and honest comparisons from the team building EvaliQA. One problem per post, from the work we do with clients.
22 posts tagged LLM judges. Clear filters
Deterministic rules, semantic similarity and LLM judges each answer a different question about an AI system. Here is how to stack them so the cheap checks carry the weight and the judge does only what nothing else can.
Read the postA metric threshold is a starting point, not a fact about your AI system. This guide shows how to label a calibration set, place the threshold, check the LLM judge rubric and separate noise from signal.
Read the postSingle-output scoring and pairwise comparison answer different questions about an AI system, and most teams pick one by habit. Here is how to choose between an absolute score and a preference, what each LLM judge pattern misses, and how to run each without fooling yourself.
Read the postA chunking or embedding change rewrites what your AI RAG system retrieves for every query at once, and spot checks will not show you what broke. This is the regression protocol we use to compare the old and new pipeline query by query before the change ships.
Read the postA CI gate for AI agent regression testing is only useful if it fails on real regressions and stays green when the LLM judge wobbles. We measure the noise floor with two identical runs, pick metrics that flip for real reasons and set the minimum pass rate with room for noise.
Read the postA custom LLM judge for an AI system can be written as one holistic rubric or as a list of discrete criteria, and the shape decides what the AI evaluation score can tell you. We explain how EvaliQA scores each, what each hides, and how to pick the shape for a given metric.
Read the postAn AI judge criterion that never references the row it scores can only judge plausibility, and in EvaliQA Custom Eval it is not judged at all. Placeholders make one AI metric read a different, concrete sentence on every row, and this post covers where the values come from and how to check them.
Read the postEvery new EvaliQA workspace now opens with the ShopWave demo project, a real support bot with planted weak spots, two test plans and two finished runs. Read the judge's reasoning on a failed row before you connect your own system.
Read the postA coding agent that edits a test to make it pass still produces an ordinary pull request. This post shows how to trace every Claude Code or Codex session into EvaliQA, score it as it arrives with detectors, metrics and the AI Analyst, and promote the bad sessions into your test dataset.
Read the postWhen a RAG answer is wrong, Faithfulness alone cannot tell you whether retrieval or generation failed. Context recall and context precision score the retrieved chunks themselves, so you fix the stage that actually broke.
Read the postAnswer Relevancy and Faithfulness get read as one quality score, but they read different columns and catch different bugs. This post shows what each one sees, what it misses, and how to read the two scores together when they split.
Read the postPromptfoo is a free open-source CLI for testing prompts and red teaming from a config file. We compare it with EvaliQA on test cases, judges, multi-turn runs, red teaming, production monitoring and reporting, and say where each one is the right pick.
Read the post
