How we evaluate AI systems, and what the metrics actually measure.
7 posts in Expertise tagged RAG. Clear filters
A chunking or embedding change rewrites what your AI RAG system retrieves for every query at once, and spot checks will not show you what broke. This is the regression protocol we use to compare the old and new pipeline query by query before the change ships.
Read the postA RAG AI system that scores well on happy-path rows can still answer a question it should decline, answer the neighbouring question, or quote a superseded policy. This guide shows how to build the three trap rows that catch each failure and how to score them in EvaliQA.
Read the postWhen a RAG answer is wrong, Faithfulness alone cannot tell you whether retrieval or generation failed. Context recall and context precision score the retrieved chunks themselves, so you fix the stage that actually broke.
Read the postAnswer Relevancy and Faithfulness get read as one quality score, but they read different columns and catch different bugs. This post shows what each one sees, what it misses, and how to read the two scores together when they split.
Read the postPrompt injection sits at the top of the OWASP Top 10 for LLM applications, and most teams test only the direct form. Here is how to test direct attacks, payloads hidden in retrieved data and multi-turn build-ups in EvaliQA, and what breach rate to accept before you ship.
Read the postA RAG answer can fail in the retriever or in the generator, and an end-to-end score cannot tell you which. Here is how the retrieval and generation metrics in EvaliQA split the two stages, and how to read them in pairs.
Read the postRetrieval failures and generation failures look identical from the outside, and averaging them into one score hides the bug. Measure the stages separately, on a dataset built to make them disagree, and read the disagreements as the diagnosis.
Read the post
