You lower the chunk size from 1,000 tokens to 400 with some overlap, or you move to a newer embedding model, and you rebuild the index. You ask the assistant a dozen questions, the answers read well, two of them look better than before, and the change ships. Three weeks later a support lead reports that the assistant no longer finds the refund terms for annual plans, and nobody connects it to the re-index because the documents did not change, the prompt did not change and no test went red. This is the problem this post is about. A chunking or embedding change touches every document in the corpus in one operation, so it cannot be reviewed by hand, and the answer-level checks most teams run are too coarse to see which queries moved. Retrieval-augmented generation (RAG) is the pattern where an AI system fetches passages from your own documents and hands them to a model that writes the answer, which means the quality of the answer is capped by the quality of what was fetched.
What we propose is a regression protocol, the same idea as a regression test suite in software but built for a retrieval pipeline. You freeze a set of queries with known relevant passages and reference answers, you run the old pipeline and the new pipeline on exactly that set, you score retrieval and generation separately with the same metrics and the same LLM judge, and you read the difference per query rather than as an average. The decision to ship is then a decision about a list of specific regressions you can see, not a feeling about a dozen spot checks. The rest of this post walks through why these changes fail silently, what each kind of change actually moves, how to build the regression set so it survives a re-chunk, which metrics to compare, the protocol itself and how to read the result, and the failures this protocol does not catch.
Why retrieval changes break silently
A fluent model hides a bad retrieval. When the right passage is missing, the model does not stop and say so; it writes a plausible answer from whatever the top-k contained plus what it remembers from pre-training. If you are reading the answer rather than the retrieved context, the failure is invisible unless you happen to know the right answer yourself. This is why the spot check passed in the opening story. The person checking asked questions they knew the general answer to, the model produced general answers, and the one query that depended on a specific clause being retrieved was not among the dozen.
The second reason is that these failures are sparse. A chunking change does not degrade every query by a little; it leaves most queries alone and breaks a few badly, usually the ones whose answer sits at a boundary that moved. If 1 query in 25 loses its relevant passage, an average answer score across the set barely moves, and a reviewer sampling 12 queries has a better than even chance of seeing none of them. Sparse, severe failures are exactly the kind an aggregate hides and a per-query comparison exposes, which is the core design choice of the protocol below.
The third reason is that nothing in the surrounding system signals the change. The corpus is the same, the generation prompt is the same, latency is similar, and the logs show retrieval returning k results as always. What changed is which k results, and that only shows up if you stored the old top-k lists and compare them to the new ones. Most teams do not store them, so the before state is gone the moment the index is rebuilt. The first practical rule of this protocol follows directly from that: never rebuild the index without first running the regression set against the old one and keeping the output.
What a chunking change moves
Chunking is the step that cuts each document into the pieces that get embedded and retrieved. Changing it changes three things at once. The boundaries move, so a fact and its qualifier can end up in different chunks; the amount of text per chunk changes, so the same k retrieves a different amount of context; and the number of chunks per document changes, so a long document that used to occupy 3 slots of the top-k may now occupy 8, crowding out other documents. Each of these has a characteristic failure. The boundary problem produces answers that are confidently incomplete, because the chunk says "refunds are available within 30 days" and the clause "except on annual plans, which are prorated" now begins the next chunk, which did not rank. The context problem produces answers that are vague when chunks are too small to carry the setup a fact needs, or answers that miss the relevant sentence when chunks are so large that the model loses it among unrelated text.
Smaller chunks tend to raise retrieval precision and lower the context the model has to reason with; larger chunks do the reverse. Neither direction is safe by default, and the trade depends on how your documents are written. Policy documents with many exception clauses punish small chunks. Reference documentation with one self-contained fact per paragraph tolerates them well. Tables are the worst case for any fixed-size chunker, because the header row that gives the columns meaning is separated from the rows, and a chunk containing "Enterprise, 99, 500, unlimited" is not retrievable for any natural question. A regression set that includes queries over tables and over exception clauses is how you find out which of these your corpus suffers from.
There is also a quiet interaction with the embedding model's input limit. If the new chunker produces some chunks longer than the model's maximum input length, those chunks are truncated at embedding time, and the truncated tail is simply not searchable. Nothing fails; the chunk is in the index with a vector that represents its first part only. Checking the distribution of chunk lengths in tokens against the model's limit before you embed is a 5-minute check that avoids a class of regressions no answer-level metric will attribute correctly.
What an embedding change moves
An embedding model maps text to a vector so that similar meanings land near each other, and retrieval is a nearest-neighbour search in that space. Replacing the model replaces the whole space. There is no sense in which a similarity score of 0.82 from the old model and 0.82 from the new one mean the same thing, so every threshold you tuned against the old distribution is void the moment you switch. If your pipeline drops results below a minimum similarity, or if a hybrid search combines vector scores with keyword scores using a weight you tuned by hand, those parameters were fitted to a distribution that no longer exists. The first regression after an embedding swap is often not the model being worse but the cutoff rejecting results it should have kept, or the hybrid weight now letting the keyword side dominate.
The new model also has its own ideas about what is similar. Some models are trained with separate instructions for queries and passages and underperform badly if you embed both the same way; some handle code, numbers or non-English text differently from the previous one; some have a different maximum input length, which brings back the truncation problem above. None of these are defects, they are differences, and the only way to know how they play out on your corpus is to measure retrieval on your queries. Benchmark scores for the new model tell you how it ranks on someone else's documents and questions.
The consequence for the protocol is that an embedding change needs to be measured at the retrieval level first, before any generation metric, and the retrieval metrics must be rank-based rather than score-based. Recall at k and reciprocal rank are comparable across two models because they ask whether the right passage was in the list and where, not how confident the model was. Comparing raw similarity scores across the swap is meaningless, and a dashboard that plots average similarity over time will show a step change that tells you nothing about quality.
Build the regression set
The regression set has three parts for each query, and all three must be frozen before either run. The first is the query itself. Draw queries from production logs rather than writing them, because the point is to protect the questions users actually ask, and stratify them so each kind of question is represented: direct lookups, questions whose answer needs 2 passages, questions about exceptions and edge cases, questions over tables, and questions whose answer is not in the corpus at all. Include every query that has previously caused an incident. A few hundred queries is a practical size for a set you will run on every index rebuild; the right number is the one where each slice has enough queries that a regression in it is distinguishable from noise, which you will find out when you run the baseline twice in step 2 of the protocol.
The second part is the gold evidence, and this is where a naive set breaks on the first re-chunk. If you label relevance by chunk identifier, the labels are meaningless as soon as the chunker changes, because the chunks no longer exist. Label the gold span instead: the document and the character range (or the sentence range) that contains the evidence for the answer. At scoring time, a retrieved chunk counts as relevant if it overlaps the gold span by more than a threshold you fix once and keep, for example if it contains the whole span or at least half of it. This mapping is cheap to compute, it works for any chunker, and it gives you a bonus signal: if a gold span now straddles two chunks under the new chunker, the mapping reports it, and that is the boundary failure from earlier made visible before anyone asks the question.
The third part is the reference answer, kept short and factual, ideally as the list of facts that a correct answer must state rather than a polished paragraph. "Refunds within 30 days; annual plans prorated instead of refunded; request via billing page" is a better reference than a well-written answer because the LLM judge in the generation step will be asked whether each fact is present, not whether the style matches. For the no-answer queries the reference is that the AI system should say it cannot find this in the documentation, and the metric is whether it abstained. Freeze the corpus snapshot too. Both pipelines must index the same documents, otherwise a difference in retrieval can be a document that was added between the runs rather than the change you are testing.
Retrieval metrics to compare
Four retrieval metrics cover most of what a chunking or embedding change can do, and they should be computed at the k you actually pass to the model, not at some larger k that flatters the result. Recall at k is the share of a query's gold spans that are covered by at least one chunk in the top-k. It answers the first question, which is whether the evidence arrived at all. Mean reciprocal rank (MRR) is 1 divided by the position of the first relevant chunk, averaged over queries, and it tells you whether the evidence arrived near the top, which matters because models attend more reliably to the beginning of the context than to the middle. Normalised discounted cumulative gain (nDCG) extends that to all relevant chunks with a position discount, and is the one to watch for multi-passage questions. Context precision is the share of the top-k that is relevant, and it drops when a chunking change floods the top-k with fragments of one long document.
Metric | What it measures | What it misses | Typical move after this change |
|---|---|---|---|
Recall at k | Whether the gold evidence is anywhere in the top-k | Position, and how much noise came with it | Falls on queries whose gold span now straddles a boundary or is truncated at embedding time |
MRR | How high the first relevant chunk ranks | Everything after the first hit | Shifts broadly after an embedding swap as the whole ranking reorders |
nDCG at k | Rank quality across all relevant chunks | Nothing about the answer; hard to explain to non-specialists | Falls on 2-passage questions when one passage drops out of the top-k |
Context precision | How much of the top-k is relevant | Whether anything relevant is missing | Falls when smaller chunks let one document fill several slots |
Top-k overlap | How many of the old top-k survive in the new top-k | Quality entirely; it measures churn, not correctness | Low after any embedding swap; tells you how many queries to inspect, not whether they got worse |
The last row is not a quality metric but it earns its place. Top-k overlap, the size of the intersection of the old and new top-k lists divided by k, tells you how much the change actually moved. An embedding swap with high overlap on most queries and low overlap on a cluster is pointing at that cluster. A chunking change with near-zero overlap everywhere is expected, since the chunks are different objects, and in that case you compare at the span level instead. Overlap is the cheapest signal in the protocol and it needs no labels, so it is also the one you can compute on thousands of unlabelled production queries to decide which ones to add to the labelled set.
Generation metrics to compare
Retrieval metrics tell you whether the evidence arrived; generation metrics tell you whether the AI system used it. Two are essential. Faithfulness asks whether every claim in the answer is supported by the retrieved context, scored by an LLM judge that is shown the answer and the chunks and asked to mark each claim as supported or not. Answer correctness asks whether the facts in the reference answer appear in the generated answer, scored by the same judge against the frozen reference. The two disagree in informative ways. A correct but unfaithful answer means the model answered from memory because retrieval failed, which is the silent failure from the start of this post, and it will stop working the day the question is about something the model does not remember. A faithful but incorrect answer means retrieval brought the wrong passage and the model dutifully summarised it, which is a retrieval regression showing up one layer down.
An LLM judge is a model prompted to score another model's output against a rubric. For a regression comparison the judge is a measuring instrument, and the rule for instruments is that nothing about them may change between the two measurements. Same judge model, same prompt, same temperature, same reference answers. Before you trust the judge on this task at all, have a person label faithfulness and correctness on a sample of answers and check that the judge agrees with them most of the time; where it does not, fix the rubric before running the comparison. Report agreement as a number alongside the results so a reader knows how much to trust a 2-point move.
Two more generation signals are worth tracking because they move in characteristic ways after these changes. The abstention rate on the no-answer slice tells you whether the new pipeline has started retrieving plausible-looking but irrelevant chunks that tempt the model into answering questions it should decline. And answer length, a crude measure, tends to climb after a move to larger chunks because the model has more material to summarise, which is not a regression in itself but is a hint that context precision has dropped. Neither replaces faithfulness and correctness; they are the cheap signals you check first when those two move.
The protocol step by step
The protocol is eight steps, and the order matters because each step fixes something the next one depends on. It assumes you have the regression set from earlier and that the change under test is one thing: a new chunker, or a new embedding model, not both. If you need to change both, run the protocol twice, because a regression you cannot attribute is a regression you cannot fix.
Snapshot the corpus and the configuration. Record the document set, the current chunker settings, the embedding model and version, the value of k, the retrieval cutoffs and hybrid weights, the generation prompt and the generation model. Both runs will use everything on this list except the one item you are changing.
Run the baseline twice. Run the regression set against the current index, then run it again. Generation is not deterministic, and some retrieval stacks are not either. The difference between the two baseline runs is your noise floor, and no move smaller than it in the comparison means anything.
Store the baseline outputs. For every query, keep the top-k chunk texts and positions, the generated answer, and the retrieval and generation scores. This is the before state that is otherwise destroyed by the rebuild.
Build the new index from the same snapshot. Apply the change, re-embed, and check the chunk length distribution against the embedding model's input limit before indexing so that truncation is caught here and not attributed to the model later.
Run the regression set on the new index. Same k, same prompt, same generation model, same judge. Store the same outputs as in step 3.
Score retrieval with the span mapping. Map gold spans onto both sets of chunks with the fixed overlap threshold, compute recall at k, MRR, nDCG and context precision for both runs, and compute top-k overlap between them.
Score generation with the frozen judge. Compute faithfulness and answer correctness for both runs, plus abstention on the no-answer slice.
Produce the per-query diff and decide against the gates. For each query, record whether it was fixed, broken, unchanged or churned, then apply the release gates you set in advance, which the next section covers.
Reading the diff
The comparison is paired. Every query has a before and an after, so the unit of analysis is the query and the first output to read is not a table of averages but four lists. Fixed queries are the ones where the gold span was missing from the top-k before and is present after, or where correctness went from failing to passing. Broken queries are the reverse. Unchanged queries moved by less than the noise floor from step 2. Churned queries are the ones where the top-k overlap is low but the scores held, which means the change rewrote the retrieval and the model coped, and which deserve a look because they are the ones most likely to break under the next change. Read the broken list first and in full. It is normally short, and every item on it is a concrete query with a concrete missing passage you can explain.
Consider a hypothetical run to see why the averages mislead. Suppose a move to smaller chunks raises recall at 5 across the whole set from 0.78 to 0.83 and leaves faithfulness flat. On the averages this is a clear improvement. The per-query diff shows that the gain comes from lookup questions, which were already mostly fine, while the exceptions slice fell from 9 of 10 queries covered to 6 of 10, because three exception clauses now begin a new chunk that ranks below the rule they qualify. Correctness on those three queries went from passing to failing, and faithfulness stayed high because the model faithfully summarised the rule without the exception. The illustrative numbers are made up; the pattern is the one we see most often after a chunk size reduction, and an aggregate recall improvement is precisely how it hides.
Release gates turn that reading into a decision. Reasonable gates for most teams are that no critical slice may lose coverage, that the number of broken queries may not exceed the number of fixed ones, that faithfulness may not fall by more than the noise floor, and that the abstention rate on the no-answer slice may not drop. A change that fails a gate is not necessarily rejected; the exceptions regression above has an obvious remedy, which is a chunker that respects the paragraph containing the exception or an overlap large enough to carry it, and the protocol is rerun after the fix. What the gate prevents is the regression shipping unnoticed. Record the diff, the decision and the reason with the configuration snapshot so that the next person who asks why the chunk size is 600 and not 400 finds the three queries that answer them.
What the protocol does not catch
The protocol measures the queries in the set, and only those. A new kind of question that users start asking after the change is not protected, which is why the set is refreshed from production logs on a schedule and why the top-k overlap check on unlabelled traffic exists. It also measures a frozen corpus. If documents are added or edited after the comparison, the new chunker will treat them in ways the set never tested, and a format the chunker handles badly, a new template for release notes for example, can arrive months after the change that introduced the weakness. Live monitoring of faithfulness on sampled production traffic is the complement to this offline protocol, and neither replaces the other.
Scale is another gap. A regression set of a few hundred queries exercises the ranking logic but not the behaviour of the approximate nearest-neighbour index at the full size of the corpus. Index parameters that trade recall for speed, such as the number of candidates examined per query, interact with the vector distribution, and a new embedding model can shift that distribution enough that the same parameters give a different approximate recall at 10 million vectors than at the 50,000 in a test index. If you can, run the protocol against an index built at production scale; if you cannot, measure approximate recall against exact search separately after the swap.
Finally, the protocol holds generation fixed and therefore says nothing about it. A change to the generation model or prompt made at the same time will confound every result, and an embedding swap bundled with a prompt tweak to "take advantage of the better context" is two experiments reported as one. Rerankers sit in the same category. If a reranker runs after retrieval, the retrieval metrics should be computed both before and after it, because a reranker can mask a retrieval regression on the labelled set and then fail to mask it on the questions the set does not contain. The protocol tells you what changed between two configurations on a fixed set of questions. That is a narrower claim than "the new pipeline is better", and it is the claim you can actually defend.
A chunking or embedding change is a retrieval experiment run on your whole corpus at once, and it deserves the controls an experiment gets. The failures it introduces are sparse and severe, they are hidden by fluent answers and by averages, and the before state disappears when the index is rebuilt, so the protocol is built around three commitments: freeze the set, the corpus and the judge before anything changes; label evidence as spans in documents rather than chunk identifiers so the labels survive the re-chunk; and compare per query against gates written down in advance. Retrieval metrics come first because they are where the change acts, generation metrics come second because they show whether the AI system compensated or failed, and the disagreement between the two is the most informative result you get. What the protocol gives you is a list of specific queries that got better and a list that got worse, which is the only form in which the question "is it safe to ship" has an honest answer.
