A retrieval-augmented generation (RAG) AI system has two parts that can fail, the retriever that pulls passages from your knowledge base and the model that writes an answer from them. Most teams evaluate it with a dataset of questions that have a clean answer somewhere in the corpus, attach Faithfulness and a correctness metric, and get a pass rate in the nineties. Then production shows the failures the dataset never contained. A user asks about a product the documentation does not cover, and the AI system answers anyway, from the model's own memory. A customer in Germany asks about returns and gets the US policy, grounded in a real passage. Someone asks about the current fee and receives last year's figure, quoted accurately from a page nobody deleted. None of these rows were in the dataset, so none of them could fail.
The problem this article addresses is a dataset that only contains answerable, well-retrieved questions. Such a dataset measures whether the AI system can answer when everything goes right, and it says nothing about what happens when retrieval returns nothing useful, something adjacent, or something out of date. We call the rows that test those cases trap rows, rows built so that the easy, fluent answer is the wrong one. This guide covers the three kinds we build for every RAG project, unanswerable, near-miss and stale-policy, how to construct each one so that it actually traps, which EvaliQA metrics score it and which do not, and how to read the results without being fooled by an AI system that passes every trap by refusing everything.
Why a happy-path RAG dataset lies
The standard RAG metrics are built around one assumption, that the right passage exists and the question is whether the AI system found and used it. Faithfulness checks that every claim in the answer is supported by the retrieved context. Contextual Recall checks that the retrieved context contains what the expected output needs. Answer Relevancy checks that the answer addresses the question that was asked. On a row where the knowledge base holds a clean answer, those three together tell you a lot, and the RAG metrics page recommends Faithfulness plus Contextual Recall plus a correctness metric as the starting set for exactly that reason. The trouble starts when the assumption behind them does not hold.
Faithfulness scores grounding, not truth. An answer that repeats a wrong passage word for word is fully grounded and scores 1.0. Answer Relevancy scores whether the response is about the question, so a confident fabricated answer to an unanswerable question is perfectly relevant. Contextual Recall needs an expected output to compare against, and on a row whose expected output is "we do not have that information" there is nothing for retrieval to recall. Each metric is doing its job. The dataset simply never asks the question these metrics would answer badly, so the AI system never gets the chance to fail in the way it fails in production.
There is a second, quieter reason happy-path datasets flatter a RAG AI system. When rows are generated from the same documents the retriever indexes, the question is often phrased in the document's own words, and the gold answer is a sentence from the passage. The docs call this out as retrieved context leaking the answer. Retrieval becomes a string match, generation becomes copy-paste, and the pass rate measures neither. Real users do not phrase questions in documentation prose, and the questions that hurt are the ones where the words match a passage but the meaning does not.
The consequence is that a 95 percent Faithfulness score on a happy-path dataset is a statement about the generator on easy rows, and nothing more. It does not tell you how the AI system behaves at the boundary of the corpus, which is where most of the production complaints come from. To see the boundary you have to put rows on it. That is what a trap row is for.
What makes a row a trap
A trap row is a single-turn dataset row whose retrieved context invites a specific wrong answer. The input is a question a real user would ask. The context, whether your AI system retrieves it live or you supply it in the row, is arranged so that the fluent, confident response is incorrect and the correct response takes some restraint, a decline, a clarification, or a choice between two passages. The expected output states the correct behaviour precisely enough that a metric can check it. The trap lives in the context, not in the question. If you make the question tricky instead, you are testing comprehension, which is a different and much better covered problem.
Trap rows use the same columns as any single-turn evaluation row. The template for a single-turn plan seeds input, expected_output and test_type, and the single-turn plan page adds retrieval_context, a JSON list of strings or plain text for one passage. We add three more columns to every trap row, a category such as trap.unanswerable, an expected_behavior value of decline, clarify or answer, and a one-sentence note explaining why the row exists. The category is what you filter on when reading a run. The behaviour column is what a custom rubric reads. The note is what stops a colleague from deleting the row in six months because the question looks odd.
There are two ways to run a trap row, and they test different things. If your AI system retrieves live and your connector maps the Retrieval context path, the LLM judge sees the passages the model actually saw, and the row tests retriever and generator together. The trap then has to exist in the real corpus, the sibling policy must really be indexed, the superseded page must really still be there. If instead you fill the retrieval_context column by hand, the metrics score against your passages, and when your system does not return passages of its own the column is what the judge uses. The datasets page is explicit that when both exist, the system's passages win. Note that the column informs the judge, not the model, so to feed frozen passages into a generation-only endpoint you bind a custom variable in the connector body to that column when you start the run. Both designs are useful. The live design finds retrieval bugs. The frozen design isolates the generator so you can tell whether the model would have fallen into the trap even with perfect retrieval.
Unanswerable rows
An unanswerable row asks a question the knowledge base cannot answer. The failure it catches is the one every RAG team eventually meets, the AI system fills the gap from the model's pretraining and presents the result in the same voice it uses for grounded answers. The user cannot tell the difference. The dangerous version is not the question about a topic the corpus has never heard of, because retrieval returns obvious noise and most prompts handle that. The dangerous version is the question about a topic the corpus almost covers, where retrieval returns passages that are on-topic and silent on the specific point.
Build the row from a real gap. Take a product or feature your documentation describes and ask about an attribute it does not state. If the docs describe a subscription plan and its price but not whether it can be paused, the input is "Can I pause my subscription for a month?" The retrieved passages will be the plan page and the cancellation page, both relevant, neither answering. The expected output is a decline that says what is missing, something like "The documentation does not say whether a subscription can be paused. It covers cancellation and plan changes, and support can confirm pausing." Write it in your product's voice, because you will score against it. Set expected_behavior to decline and the note to "Pausing is not documented; cancellation page is a near-match."
Scoring an unanswerable row takes two metrics that look at the response from opposite directions. Faithfulness catches the fabrication, because a claim such as "yes, you can pause for up to three months" has no support in the context and pulls the score below the 0.7 default threshold. Restricted Refusal, a heuristic metric that runs without an LLM judge, checks whether the response reads as a refusal and scores 1 when it does. Between them you can tell three outcomes apart, a fabricated answer (Faithfulness fails), a bare "I can't help with that" (Restricted Refusal passes, which is only half the job), and a decline that names the gap and points to the next step, which is what you want. For the third outcome add a Custom Eval criterion, "The response states that the available information does not cover the question, and matches the reference answer {{expected_output}} in substance."
What this row does not catch is the opposite failure, an AI system that declines when the answer was in the context. A prompt tuned hard against hallucination will drift toward "please contact support" on questions it could have answered, and unanswerable rows will reward that drift. The only defence is to run the happy-path dataset alongside the trap dataset and read both numbers together, which we return to below.
Near-miss rows
A near-miss row gives the AI system the answer to a neighbouring question. The passage is real, it is on-topic, and it is wrong for this user. The classic shape is a rule that varies by segment, region, plan tier, product version or customer type, where the corpus holds one page per variant and retrieval returns the wrong variant because the wording overlaps almost entirely. A US customer asking "How long do I have to return an item?" retrieves the EU returns page, which says 14 days, when the US page says 30. The model answers "14 days", every claim is supported by the context, and Faithfulness scores 1.0.
This is the trap row Faithfulness cannot catch by design, and it is the reason a correctness metric belongs on every RAG plan even when you trust the grounding. Which correctness metric matters. Answer Precision compares the response with the expected output as text, with a tolerance for numbers, and makes no judge call, so it works when the expected output is the bare value, "30 days". Semantic Similarity compares meaning with an embedding model, and the deterministic metrics page gives the number that should worry you here, "Refunds are available within 30 days" and "Refunds are not available within 30 days" score 0.93 against each other. A near-miss answer differs from the correct one by a single fact, and that is exactly the difference an embedding similarity score smooths over. For free-text expected outputs, use a Custom Eval criterion that names the fact, "The response states the return window for a United States customer as 30 days, matching {{expected_output}}, and does not state 14 days."
Build near-miss rows from sibling documents. List every rule in your corpus that exists in more than one variant, then write one input per variant that states the user's segment plainly but not in the document's words. "I'm in Ohio, how long for a return" is better than "What is the United States return window", because the second phrasing matches the right page by keyword and the trap never springs. Fill expected_output with the value for that segment. If you run with live retrieval, also attach Contextual Relevancy, which judges whether the retrieved passages are on-topic for the question. A low Contextual Relevancy score alongside a wrong answer tells you the retriever picked the sibling; a high score alongside a wrong answer tells you the retriever returned both variants and the model picked badly, which is a prompt problem rather than an index problem.
A second kind of near-miss is worth a few rows, the question where the user has not given enough to choose a variant. "How long do I have to return an item?" with no region is not answerable with a single number, and the correct behaviour is to ask which region or to state both values with their conditions. Set expected_behavior to clarify and write the criterion accordingly. An AI system that answers "30 days" here is guessing, and the docs note that an AI system that guesses right on the dataset will guess wrong on the row where guessing does not land.
Stale-policy rows
A stale-policy row puts two versions of the same rule in front of the AI system. Knowledge bases accumulate. A pricing page is updated and the old one is archived but still indexed. A policy changes on a date and the announcement sits next to the policy it replaced. A versioned API has docs for v1 and v2 under similar headings. Retrieval has no notion of time, so both passages come back, and the embedding of the older passage is often closer to the question because it has been quoted more, linked more and phrased more plainly. The model answers with the superseded value, accurately, and Faithfulness, Contextual Relevancy and Answer Relevancy all pass.
Build the row from a real change in your own corpus. Find a rule that changed, keep both documents indexed as they are in production, and ask for the current value with a phrasing that does not include the date, "What's the monthly fee on the Team plan?" The expected output is the current value together with the date it took effect, "The Team plan is 49 dollars a month, as of 1 March 2026. Before that it was 39 dollars." Including the date in the expected output does two things, it gives the LLM judge a concrete thing to check, and it makes the row self-documenting when someone reads it next year. If you supply the context by hand, put both passages in retrieval_context in the order your retriever actually returns them, old first if that is what happens, because order is part of the trap.
Two metrics do the work here. A Custom Eval criterion names the current value and forbids the old one, "The response gives the Team plan fee as 49 dollars a month, matching {{expected_output}}, and does not present 39 dollars as the current fee." The second is Contextual Precision, which scores whether the most useful passages are ranked first and takes an optional top_k so it reflects only the chunks the model actually received. On a stale-policy row the most useful passage is the current one, so a low Contextual Precision score alongside the wrong fee tells you the retriever ranked the archive above the live page. The docs warn that Contextual Precision needs ranked context, and a column that stores passages as an unordered set turns it into noise. That is why order matters when you fill the column by hand.
Stale-policy rows age differently from the other two kinds. When the policy changes again, the expected output is wrong, and the row starts failing a correct AI system. Give every stale-policy row a last_reviewed date and include them in the quarterly review the datasets page recommends. Use Duplicate on the dataset before you update a batch of them, so the runs that used the old expected outputs stay reproducible against the frozen version. A stale trap row that nobody maintains becomes the thing it was built to catch.
How to score trap rows in EvaliQA
Trap rows do not belong in the same dataset as happy-path rows. A run's pass rate is only readable when the denominator is homogeneous, and the dataset editor warns when a generation mixes types for this reason. "92 percent on happy path" and "100 percent decline on unanswerable" tell you two things; "94 percent on a mix" tells you nothing, and it hides an AI system that answers every unanswerable question behind a large happy-path majority. We keep one dataset per trap type, attached to the same test plan as the happy-path set so the judge model, the thresholds and the connector are held constant. The steps below assume a single-turn plan that already runs against a happy-path dataset.
Open the test plan and create a new dataset from the single-turn template, or upload a CSV whose header has
input,expected_output,retrieval_context,category,expected_behaviorandnote. Uploaded columns arrive as text, so retyperetrieval_contextasjsonin the column menu if you store a list of passages.Write the rows by hand. Trap rows are golden rows in the sense of the golden dataset page, every one exists for a reason, and the generator cannot know which rule in your corpus has a sibling or a superseded version.
On the plan's Metrics step, keep Faithfulness and the correctness metric you already use, add Restricted Refusal for the unanswerable dataset and Contextual Precision for the stale-policy dataset, and add a Custom Eval with the criteria described above. Turn on Every criterion must pass on the Custom Eval, because a trap row criterion is a rule, not a quality rubric.
If the Custom Eval references a custom column such as
{{expected_behavior}}, tick that column under Extra fields when you start the run. A criterion whose placeholder does not resolve is dropped silently and listed under skipped criteria, and a row with no criterion left shows n/a, which does not count toward the pass rate.If your AI system does its own retrieval, map the Retrieval context path in the connector so the judge scores the passages the model saw. If you are feeding frozen passages to a generation-only endpoint, bind a custom variable in the connector body to the
retrieval_contextcolumn when the New eval run sheet asks.Run the trap datasets and the happy-path dataset against the same plan, and compare per-category pass rates in the run's row list.
Two configuration details decide whether the trap works. The first is the judge's default verdict. An LLM judge scoring Answer Relevancy will treat "the documentation does not cover this" as a poor answer, because it does not address the question, and Contextual Recall has nothing to recall when the expected output is a decline. Leave those two off the unanswerable dataset, or accept that they report failures that are not failures. The eval types page makes the same point for forbidden-topic rows, a refusal there is a pass, and the metric has to know it. The second is the Faithfulness threshold. The default is 0.7, which lets an answer with one unsupported claim in four pass. On trap datasets we set it to 0.9 or above, in line with the docs' advice for compliance-sensitive products, because the single unsupported claim is the entire point of the row.
Trap rows pay off in the combinations of scores, not in any single metric. On an unanswerable row, Faithfulness failing means the AI system fabricated, while Faithfulness passing with Restricted Refusal failing means it produced a grounded but non-committal answer that neither declined nor invented, which is usually a prompt that says "answer from the context" without saying what to do when the context is silent. On a near-miss row, Faithfulness passing with the Custom Eval failing is the signature of the trap springing, and Contextual Relevancy tells you whether to fix the index or the prompt. On a stale-policy row, the Custom Eval failing with Contextual Precision low points at ranking, and the same failure with Contextual Precision high means the model had the current passage in front of it and chose the old one anyway, often because the old passage states the value more plainly.
Read the judge's reasoning on every failed trap row before you change anything. The bottleneck page recommends filtering to failed rows and clustering by the reason, and trap rows cluster cleanly because each dataset isolates one failure. Ten unanswerable failures that all cite the same fabricated attribute are one prompt bug. Ten near-miss failures that all involve the same pair of sibling pages are one chunking or metadata bug, and the fix is likely to add the segment as a filterable field at index time rather than to touch the prompt at all. Ten stale-policy failures on the same archived page are one missing deletion.
Compare runs only when the trap datasets are frozen. Trap rows are hand-written and small, so the temptation is to edit them in place after a run reveals a weak expected output. Use Duplicate first. The pass rate on the trap datasets is one of the numbers you will put in the release report, and the comparing runs page is clear that a pass-rate delta on a moving dataset means nothing. Hold the datasets, the judge model and the target model constant, and let the only variable be the change you made to the AI system.
Where trap rows come from and how many
Trap rows are small in number and expensive in thought. A good one takes the 3 to 10 minutes the golden dataset page quotes for a hand-written row, and most of that time goes into finding the gap, the sibling or the superseded page in your own corpus. For a new RAG project we start with 5 to 10 rows per trap type, which fits comfortably inside the 10 to 50 row golden set the docs recommend. For a mature product with many segmented rules and a long document history, 20 to 30 per type is still manageable, and above that the rows stop being individually justified. The question to ask of every candidate is whether it traps a distinct way of being wrong. Two near-miss rows on the same pair of sibling pages are one test counted twice.
The best source after your own reading of the corpus is production. Every support ticket that says "the assistant told me X and it was wrong" is a near-miss or stale-policy row with the expected output already written by the support agent who corrected it. Every escalation where the AI system should have said it did not know is an unanswerable row. Once runtime tracing is wired in, the trace page's Add to dataset action promotes the real question into the trap dataset, and you add the expected output, the category, the behaviour and a note citing the ticket. Tag the source as production or incident, keep the row after the fix ships, and the trap dataset becomes the regression alarm for every grounding failure a real user has ever found.
What the generator can and cannot do here is worth being precise about. The Generate action writes rows from the project description, scenarios and uploaded documents, and it will happily produce questions outside the uploaded scope that look like unanswerable rows. Review them, because the generator does not know which of those questions your production corpus actually answers, and an unanswerable row whose answer is in the index is a happy-path row with a wrong expected output. It cannot produce near-miss or stale-policy rows at all, because it does not know which rules have siblings or which pages are superseded. Those come from someone who knows the corpus.
