The support bot tells a customer their refund window is 14 days. The policy says 30. The sentence is fluent, confident, correctly formatted, and wrong, and nothing in your logs marks it as different from the thousands of answers that were right.
This is the characteristic failure of a retrieval-augmented system, and it is why a single "answer quality" number is not enough to run one. The number tells you something went wrong. It does not tell you where, and without that, the next two days are spent guessing at the prompt.
The same symptom, three different bugs
A wrong answer from a RAG pipeline has three plausible causes, and they call for three unrelated fixes:
| Where it broke | What you see | What fixes it |
|---|---|---|
| Retrieval | The right passage was never fetched | Chunking, embeddings, query rewriting, the index |
| Ranking | It was fetched, but sat at position 14 | Reranking, top-k, hybrid search |
| Generation | It was right there in context, and the model ignored it | Prompt, model, output constraints |
Three different teams' worth of work, and one score cannot distinguish them. This is the whole argument for stage-level metrics.
Measure retrieval without asking the model anything
Retrieval quality is answerable before generation runs at all, which makes it the cheapest signal in the pipeline.
- Context recall. Of the facts needed to answer, how many appear in the retrieved context? This is the one that catches a chunking bug.
- Context precision. Of what was retrieved, how much was relevant? Low precision does not always break the answer, but it burns tokens and gives the model more ways to go wrong.
- Rank of the first relevant passage. A number, not a judgement. If it drifts upward after an index change, you know before anyone complains.
None of these needs a judge. They need a dataset where somebody wrote down which document holds the answer, which is a morning of work and the highest-leverage morning in the project.
Then measure generation, given the context it got
Only once retrieval is accounted for does it make sense to grade the answer, and then the question is narrow: given exactly this context, is the answer supported?
- Faithfulness. Every claim in the answer traceable to the context. This is the one that catches the polite invention.
- Answer relevancy. It addresses what was asked, rather than an adjacent question it found easier.
Run these conditioned on the retrieved context, not on the ground truth. A generation metric that punishes the model for context it never received is measuring retrieval a second time, badly.
The trap rows are the point
A dataset of questions your system answers well tells you nothing you did not already believe. The rows that earn their place are:
- Unanswerable questions, where the corpus genuinely does not contain the answer. The correct behaviour is to say so. Most systems invent.
- Near-miss questions, where a very similar document exists and is wrong. This is where ranking failures hide.
- Multi-hop questions, needing two documents. Recall looks fine on each half and the answer is still wrong.
- Stale-policy questions, where the corpus holds both the old and the new version. Whichever ranks higher wins, which is rarely what you want.
Twenty rows of this kind are worth two hundred happy-path rows.
The regression is what you are really buying
None of this is interesting once. The value arrives the fourth time you change the chunk size and can say, in ten minutes, that recall went up, precision went down, faithfulness held, and the near-miss rows got worse. That sentence is a decision. A single number is an argument.
