Metamorphic testing (MR1–MR8)

Metamorphic testing is a technique borrowed from classical software testing: instead of asserting "the answer must equal X", you assert "the answer must have relation R to the answer for a paraphrased version of the same question". It sidesteps the hardest part of LLM evaluation, you don't need a gold answer, just a claim about how the answer should change (or not change) when the input changes.

In practice this catches a specific and important class of bug: prompt fragility. A model that answers correctly when the user types "How do I refund?" and confidently wrong when the user types "I need my money back" is not a system you can rely on. Metamorphic tests find that instability.

EvaliQA generates metamorphic pairs for eval plans when you tick Metamorphic testing as a dataset type or pick from the Metamorphic relations picker in the generator. Eight relations are available.

MR1: Synonym replacement

What changes. Key words in the input are swapped for synonyms.

Expected behaviour. Output stays equivalent, same meaning, comparable structure, comparable factual claims.

Example.

• "What's your refund policy?" → "What's your return policy?"
• "Can I cancel my order?" → "Can I annul my order?"

When to use. For any product where users might reach for different vocabulary, customer support, retail, medical, legal. Fails often signal over-fitted retrieval keyword matches or brittle intent classifiers upstream of the LLM.

Metric. A G-Eval or Custom Eval rubric that checks the two paired answers convey the same fact. EvaliQA doesn't ship a built-in similarity metric, so the paired judgment is authored as a custom rubric.

MR2: Paraphrasing

What changes. The whole sentence is rewritten preserving meaning.

Expected behaviour. Meaning and output are preserved.

Example.

• "How do I track my package?" → "Where can I check my order's delivery status?"

When to use. As a broader superset of MR1. Paraphrasing tests both lexical and syntactic robustness. This is the one you want if you can only afford one MR type.

Metric. Same as MR1.

MR3: Adding context

What changes. Extra, relevant context is added to the input.

Expected behaviour. Output remains correct, ideally improves, but must not degrade.

Example.

• "Can I refund?" → "I bought a jacket last week (order #12345). Can I refund?"

When to use. For RAG and agent systems. Fails often reveal that the model latches onto irrelevant tokens or gets confused by extra information, a warning sign for real-world deployments where inputs are rarely minimal.

Metric. Correctness must not drop on the enriched input. A drop under MR3 is a strong signal to look at retrieval / context handling.

MR4: Negation

What changes. A condition in the question is negated.

Expected behaviour. The output reflects the logical flip.

Example.

• "Which products are in stock?" → "Which products are out of stock?"
• "Refund policy for members" → "Refund policy for non-members"

When to use. For any system that answers questions about state or policy. Failure signals the model is pattern-matching on nouns and ignoring the semantics of the qualifier, a common and dangerous mode.

Metric. A Custom Eval rubric that checks the negated output correctly inverts the original claim. Don't reach for a "similarity"-style rubric here, for negation the answers should differ, so equivalence-scoring is the wrong shape.

MR5: Focus shift

What changes. The focus of the question is shifted while the topic stays the same.

Expected behaviour. The answer follows the new focus.

Example.

• "Tell me about the return policy" → "Tell me about the return window specifically"

When to use. For assistants that answer open-ended questions. Failure shows the model returns the same canned answer regardless of what the user emphasised.

Metric. Custom LLM judge that verifies the shifted focus is addressed in the output.

MR6: Compound question

What changes. Two related questions are merged into one input.

Expected behaviour. All parts of the compound question are answered.

Example.

• "What's your refund policy?" + "How long does a refund take?" → "What's your refund policy and how long does a refund take?"

When to use. For chat products where users pack multiple asks into one turn. Failure typically means the model answers only the first part and drops the rest.

Metric. A custom LLM judge that scores whether both sub-questions are addressed in the output.

MR7: Ambiguity

What changes. The input is made deliberately ambiguous.

Expected behaviour. The model either asks for clarification or handles the ambiguity gracefully (e.g. covers both interpretations).

Example.

• "Cancel my order" (which order? there are three)
• "What's the price?" (of what?)

When to use. For chatbots and voice bots, real users produce ambiguous inputs constantly. Failure = the model guesses confidently without asking, which is worse than asking.

Metric. Custom judge: was clarification asked, or were both/all interpretations covered? Confidently picking one interpretation without disclaiming is a fail.

MR8: Partial information

What changes. Details are omitted from the input.

Expected behaviour. The model still produces a reasonable answer, either a general one or one that asks for the missing detail.

Example.

• Full: "How do I refund order #12345 from last week paid by Visa?"
• Partial: "How do I refund an order?"

When to use. For agents that call tools with structured arguments. Failure often means the model hallucinates the missing values instead of asking or falling back to a general answer, potentially catastrophic if a tool call is fired with fabricated arguments.

Metric. Custom judge that penalises hallucinated specifics. A graceful "I need the order number to proceed" is a pass.

Reading the results

Metamorphic tests produce paired rows in the run's result list. When you drill into a pair:

• Compare the two outputs side by side.
• If the metric fired, its verdict tells you whether the pair honoured the relation.
• A pattern of failures on one MR type, say, ten failures on MR4 (negation), is a structural signal: the model is missing that specific reasoning axis and no amount of prompt tweaking will fix all of it. Usually the fix is either a stronger model or a rewriting layer that normalises inputs.

Composing a metamorphic mini-dataset

A useful metamorphic pass isn't huge. A recipe that works well:

• Take 10–20 representative happy-path inputs.
• Generate pairs across 3–4 MR relations that matter most for your product (typically MR2 Paraphrasing, MR4 Negation, MR7 Ambiguity, plus MR8 Partial information for agents).
• You end up with ~60–100 rows total, enough to spot patterns, cheap enough to run on every release.

Tips and pitfalls

• Don't reuse an equivalence rubric on MR4 or MR6. The whole point of those relations is that the outputs should differ (a negation flips the answer; a compound question expands it). Use a Custom Eval rubric shaped for the relation instead.
• A "pair" needs to be a real paraphrase, not a re-typing. Adding a period doesn't count. Vary the words and structure meaningfully.
• Metamorphic tests don't replace correctness tests. They tell you whether the model is consistent, not whether it's right. Combine metamorphic with a happy-path correctness metric for a complete picture.
• Metamorphic rows can inflate cost. Every pair is 2 (or more) LLM calls plus judge calls. Keep the pool modest, 60–100 pairs is usually enough.
• Watch for asymmetric failures. If output A is correct and B is wrong, the pair fails, but which of the two is "correct" depends on the relation. For MR1/MR2 they should agree; for MR4/MR6 they should differ in a specific way.

Related

• Datasets: methodology hub
• Evaluation dataset types
• Metrics