Every LLM judge starts with a decision that gets less attention than the judge prompt or the choice of model behind it. You can ask the judge to look at one AI output and give it a score against a standard, or you can show it two AI outputs and ask which one is better. These are the two basic judge patterns, single-output scoring and pairwise comparison, and they produce different kinds of evidence. A score of 4 out of 5 tells you where an output sits against a rubric. A preference for output B over output A tells you nothing about where either sits, only that one beat the other. Teams tend to pick whichever pattern they met first and then apply it to every evaluation job they have, from regression testing to model selection to live monitoring.
That is the problem this post is about. The pattern you choose decides what question your evaluation can answer, what it will miss, and how it will fail when the judge is wrong. Single-output scores drift as the judge's internal sense of a 4 shifts between runs and model versions. Pairwise verdicts flip with the order of the answers and reward length and style over substance. Neither failure is fatal, but each needs a different fix, and neither pattern can substitute for the other on every job. We walk through what each pattern measures, what the research says about their reliability, the specific ways they break, and how we match them to the jobs in a test plan at EvaliQA.
What single-output scoring is
In single-output scoring the LLM judge receives one input, one AI output, and a rubric, and returns a score on a fixed scale. The scale can be binary, pass or fail, or a short ordinal scale such as 1 to 5, or a label from a fixed set. The rubric says what each level means. A faithfulness metric for a retrieval-augmented AI system might define a 5 as every claim in the answer being supported by the retrieved passages, a 3 as most claims supported with one unsupported detail, and a 1 as the answer contradicting the passages. Reference-based variants add a gold answer written by a human and ask the judge to score the output against it, which turns a vague quality judgement into a narrower comparison task.
The output of this pattern is an absolute number per test case, and that is its main appeal. You can average the scores across a dataset, track the average across releases, set a threshold for a release gate, and break the scores down by category to find where the AI system is weak. If your customer-support AI agent scores 4.2 on faithfulness this week and 3.6 next week, that is a signal that something changed, and you can find the cases that dropped. Nothing else in the evaluation needs to exist for a single-output score to be meaningful. There is no baseline version, no competitor model, no second output to compare against.
The cost of that independence is that the judge has to carry the standard in its head. A 4 only means something if the judge gives a 4 to the same kind of output every time, on every run, and after the judge model is updated. That is a calibration problem, and it is where this pattern fails. The rubric helps, reference examples for each score level help more, but the judge is still mapping a continuous sense of quality onto a few discrete levels, and the boundaries between levels are the judge's to place. Two outputs that a human would rank clearly, one a little better than the other, can both land on 4 because the gap between them is smaller than the width of a level. Single-output scoring is good at telling you whether an output is acceptable and poor at telling you which of two acceptable outputs is better.
What pairwise comparison is
In pairwise comparison the LLM judge receives one input and two AI outputs, usually labelled A and B, and returns a preference, A, B or tie, often with a short justification. The judge is no longer asked where each output sits on a scale, only which one better meets the criteria in the prompt. This is the pattern behind preference leaderboards such as Chatbot Arena, where many pairwise votes are aggregated into a ranking, and behind most A/B comparisons of prompts, models and fine-tunes. The paper that formalised the term LLM-as-a-judge, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, studied both patterns and reported that a strong judge reaches over 80 percent agreement with human preferences, the same level that humans reach with each other.
The judge's task is easier here than in single-output scoring, and that is the main appeal. The judge does not need a stable internal standard, it only needs to notice which of two concrete answers is better on the stated criteria. Relative judgements are easier for people too, which is why human preference data is usually collected pairwise rather than as absolute ratings. The verdicts aggregate into a win rate, the share of test cases on which the candidate beat the baseline, and a win rate of 62 percent against the current production prompt is a decision-ready number in a way that a 0.3 point rise in average score is not. When many candidates need ranking, the pairwise verdicts feed a rating model such as Bradley-Terry or Elo that turns preferences into a single ordered list.
The cost is that a preference says nothing on its own. Pairwise comparison always needs a second output, so it needs a baseline version to compare against or a set of candidates to rank, and it produces nothing you can monitor over time without re-running the comparisons against a fixed reference. The number of comparisons also grows quickly. Comparing one candidate to one baseline over 500 test cases is 500 judge calls, or 1,000 with the order-swap we describe later. Ranking 8 candidates against each other on the same cases is 28 pairs per case, 14,000 calls, or 28,000 with the swap. And the pattern has a blind spot that single-output scoring does not have. If both outputs are wrong, the judge will still pick one, and a 70 percent win rate over a bad baseline tells you nothing about whether the winner is good enough to ship.
What the research says about each pattern
The evidence on reliability points in two directions at once, and understanding why is the key to choosing well. On agreement between judges, pairwise comparison wins. WebDevJudge, a 2025 study of LLM judges on web development tasks, found that pairwise comparison raised agreement between different judge models by an average of over 8 points compared with single-answer grading. The authors attribute the gap to the cognitive load of applying a multi-point Likert scale consistently, which demands a calibrated internal standard of quality that both humans and LLMs find hard to maintain. In their data some judge models even ordered the systems under test differently under single-answer grading than under pairwise comparison, which means the absolute scores were not just noisier but sometimes wrong in rank order.
On resistance to being gamed, single-output scoring wins. Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation tested what happens when a generator model learns to exploit spurious attributes the judge happens to like, which the authors call distractor features, such as a confident tone or a longer answer. Under those conditions pairwise preferences flipped in about 35 percent of cases, while absolute scores flipped in about 9 percent. The authors' reading is that absolute scoring produces judgements that better reflect response quality and are less influenced by distractors, and they recommend choosing the protocol based on the dataset and the evaluation objective rather than treating either as universally better.
These results are not in conflict once you see what each measures. Pairwise comparison is an easier task for the judge, so independent judges agree more often, but the judge is looking for any difference between two answers, and a difference in style is a difference. Single-output scoring is a harder task, so judges agree less often, but the rubric anchors the judge to the criteria you wrote and a stylistic flourish has nowhere to go if the rubric does not reward it. The practical consequence is that pairwise comparison gives sharper verdicts on the dimension you care about and also on dimensions you do not, while single-output scoring gives duller verdicts that stay closer to the rubric. Which trade you want depends on whether the AI outputs under test were optimised with the judge in the loop, which is exactly the situation in which distractor features appear.
Position bias and other pairwise failures
The MT-Bench paper named three biases in LLM judges that have been confirmed many times since. Position bias is the tendency to prefer whichever answer appears first or second regardless of content. Verbosity bias is the tendency to prefer the longer answer. Self-enhancement bias is the tendency of a judge to prefer outputs written by a model from its own family. All three show up most sharply in pairwise comparison, because that is the pattern in which the judge is choosing between two things and any lean, however slight, decides the verdict. The authors proposed swapping the order of the two answers and running the judge twice as a mitigation, and found that it reduces position bias without eliminating it, since a judge whose preference depends on order will contradict itself across the two runs rather than converge.
A later study, Judging the Judges, A Systematic Study of Position Bias in LLM-as-a-Judge, measured the effect across 15 judge models, 2 benchmarks, 22 tasks and roughly 40 answer-generating models. It introduced three metrics, repetition stability for whether a judge gives the same verdict when the same prompt is repeated, position consistency for whether the verdict survives swapping the order, and preference fairness for whether the judge leans toward one position overall. Its most useful finding for practitioners is that position bias is strongly affected by the quality gap between the two answers and only weakly by the length of the prompt components. When one answer is clearly better the judge picks it in either order. When the two are close the order decides, and close pairs are exactly the ones you run a comparison to settle.
That finding tells you how to read a flipped verdict. If the judge picks A when A is first and B when B is first, the honest interpretation is a tie, not a coin flip to be resolved by averaging. Treating order-inconsistent pairs as ties shrinks your win rate toward 50 percent, which feels like losing signal, but it is removing noise that was never signal. Length bias needs a different fix. You can instruct the judge to ignore length, which helps a little, or you can report the median length of each candidate's outputs next to the win rate so a reader can see when the winner is simply longer. Self-enhancement bias is handled by choosing a judge from a different model family than the AI system under test, or by using two judges from different families and only trusting verdicts they agree on. What none of these fixes catch is the case where both outputs are bad, and for that you need a single-output score alongside the comparison.
Calibration drift in single-output scoring
Single-output scoring fails more quietly. The judge does not contradict itself in a way you can see from one run, it shifts its standard slowly, and the shift shows up as a trend in your metric that has nothing to do with the AI system. The most common cause is a change in the judge model. If your faithfulness metric used a judge model that was updated or replaced, the mapping from quality to score moved with it, and a drop from 4.2 to 3.8 across the update is not a regression in the AI system. The second cause is the scale itself. A 1 to 10 scale gives the judge 9 boundaries to place and it places them differently on different days. A 1 to 5 scale gives it 4. A pass or fail decision with a clear rubric gives it 1, and the paper on judgement distributions cited above found that reasoning before scoring can increase the spread of scores rather than reduce it, so a longer judge prompt is not automatically a steadier one.
The fix is to anchor the scale and then measure the anchor. Anchoring means writing the rubric with a concrete example output for each score level, drawn from your own data, so the judge is matching to exemplars rather than interpreting adjectives. A rubric that says a 3 is "partially faithful" invites drift. A rubric that shows an actual answer with one unsupported claim and labels it a 3 does not. Measuring the anchor means keeping a frozen set of 100 to 200 outputs with human-assigned scores, scoring it with the judge every time the judge prompt or judge model changes, and reporting the agreement with the human labels. If agreement drops, the judge moved, and any trend in the live metric across that change is suspect until the judge is re-anchored. Pinning the judge model version in the test plan is the simplest form of this discipline, and the one most teams skip.
Even a well-anchored single-output metric does not catch small differences between two versions of an AI system. If the current prompt and the candidate prompt both produce answers that land on 4, the score is identical and the metric says nothing, although a human reading the two side by side would prefer one. This is the mirror image of the pairwise blind spot. Pairwise comparison cannot tell you whether an output is acceptable, single-output scoring cannot tell you which of two acceptable outputs is better, and a test plan that needs both answers needs both patterns.
Matching the pattern to the evaluation job
The question that settles the choice is "compared to what?" If the honest answer is a standard, a rubric that says what good looks like regardless of what else exists, single-output scoring is the pattern. If the honest answer is another output, the current version, a competitor model, a different prompt, pairwise comparison is the pattern. Most evaluation jobs have a clear answer once the question is put this way, and the table below lists the ones that come up most often in test plans we build at EvaliQA.
Evaluation job | Pattern | Why | What you still need |
|---|---|---|---|
Regression testing across releases | Single-output | Needs a number that means the same thing in every run | Pinned judge version, anchored rubric |
Release gate with a threshold | Single-output | A gate is a comparison to a standard, not to a version | Human-labelled set to validate the threshold |
A/B test of two prompts | Pairwise | The question is which is better, and the differences are small | Order swap, length reported |
Choosing between candidate models | Pairwise | Relative ranking, judges agree more on it | A judge from a different model family |
Live monitoring of production traffic | Single-output | There is no second output to compare in production | Sampling, drift check on the judge |
Ranking many candidates | Pairwise with a rating model | Preferences aggregate into a ranking | Budget for the number of pairs |
Red teaming an AI system | Single-output | An attack either succeeded or it did not | Binary rubric, human review of failures |
Two of these rows deserve a word more. Live monitoring is single-output by necessity, since the AI system produced one answer for a real user and there is nothing to compare it against, and that is why calibration drift matters most there. A monitoring metric that drifts with the judge will raise alerts for changes that never happened or hide changes that did. Red teaming is single-output because the criterion is binary and absolute. A jailbreak attempt either produced disallowed content or it did not, and asking a judge which of two attack responses is "more harmful" is rarely the question anyone needs answered. Where the table says pairwise, the comparison is almost always against the current production version, because the decision the evaluation feeds is whether to replace it.
The jobs that are hard to place are the ones where you want both answers at once, such as a model migration where you need to know that the new model is at least as good as the old one and also that it clears the absolute bar the old one cleared. The honest approach is to run both patterns and report both, a win rate against the current version and a score against the rubric, and to decide on the two numbers together. A candidate that wins 58 percent of comparisons but scores below the release threshold is better than what you have and still not good enough, and only the pair of numbers shows that.
How to run each pattern well
The setup for each pattern is short, and the steps that matter are the ones that guard against the failure mode of that pattern. For single-output scoring the work is in the rubric and the anchor set. For pairwise comparison the work is in the order swap and in choosing what to compare against. Here is the sequence we follow for a single-output metric.
Write the rubric with a concrete example output for each score level, taken from the AI system's own outputs, and keep the scale short, pass or fail where the criterion allows it and 1 to 5 at most.
Build a frozen anchor set of 100 to 200 outputs with human-assigned scores that cover every level of the scale, including the levels the AI system rarely produces.
Score the anchor set with the judge and report agreement with the human scores before trusting the metric on anything else.
Pin the judge model version in the test plan and re-run the anchor set whenever the version or the judge prompt changes.
Report the score distribution, not only the mean, so a shift in the share of 1s is visible even when the average barely moves.
For a pairwise metric the sequence is different, and most of it is about making the verdicts trustworthy before aggregating them.
Fix the baseline, usually the current production version, and keep it fixed for the whole comparison so the win rate has a stable meaning.
Run every comparison twice with the order of the two outputs swapped, and record a tie whenever the two verdicts disagree.
Choose a judge from a different model family than either output's generator, or use two judges and count only the verdicts they share.
Report the median output length of each side next to the win rate so a length-driven result is visible.
Read a sample of the justifications by hand, at least 30, to check that the judge is preferring on the criteria you wrote and not on tone or formatting.
Both sequences end with a human check, and that is not optional. The agreement figures in the papers above are averages over tasks, and the agreement on your task with your rubric is something you have to measure. A judge that agrees with humans 85 percent of the time overall can agree 60 percent of the time on the one category that matters to you, and the only way to find out is to label a sample and compare.
Using both patterns together
The two patterns check each other, and the simplest way to use that is to turn single-output scores into implied preferences and compare them with pairwise verdicts on the same cases. If output A scored 4 and output B scored 3, single-output scoring implies A is preferred. If the pairwise judge picked B, the two patterns disagree on that case, and the disagreement is worth reading. The MT-Bench authors did this conversion to measure agreement between the two modes, and in a test plan it is a cheap consistency check that needs no extra labelling. Cases where the patterns disagree are usually cases where the two outputs are close in quality, where the rubric does not capture what makes one better, or where a distractor feature swayed the pairwise verdict, and each of those is a different thing to fix.
A more deliberate combination is to assign each pattern a role. Single-output metrics run on every release and on a sample of live traffic, and they answer the question "is the AI system still acceptable?" Pairwise metrics run when a change is proposed, and they answer the question "is the candidate better than what we have?" The release decision reads both. A candidate must win the pairwise comparison against the current version and must clear the single-output thresholds on its own, and either condition failing blocks the release. This keeps the pairwise blind spot, two bad outputs, covered by the absolute score, and keeps the single-output blind spot, two outputs that both land on 4, covered by the comparison.
There is a third combination that is useful when you suspect your pairwise judge is rewarding style. Score both outputs with a single-output rubric that excludes style, for example a faithfulness rubric that only checks claims against sources, and then look at the pairwise win rate within each score band. If the candidate wins 70 percent of comparisons overall but only 52 percent among pairs where both outputs scored the same on faithfulness, most of the win rate came from something other than faithfulness. That might be fine, if the something is clarity, or it might be the distractor effect the pointwise-versus-pairwise paper measured, and you will not know which without reading the justifications. Either way the combination tells you what the pairwise number is made of, which a win rate on its own never does.
What to take away
The choice between pairwise comparison and single-output scoring is a choice about what question the evaluation answers, and the two questions are not interchangeable. Single-output scoring compares an AI output to a standard and gives you a number you can threshold, trend and monitor, at the price of a judge that has to hold the standard steady across runs and model versions. Pairwise comparison compares an AI output to another output and gives you a sharper verdict that independent judges agree on more often, at the price of order and length effects and a result that says nothing about whether either output is good enough. Research on both patterns supports both halves of that trade, with pairwise raising inter-judge agreement by over 8 points in one study and absolute scoring flipping on distractors a quarter as often in another.
In practice we build test plans that use both and assign them roles. Single-output metrics with anchored rubrics and a frozen human-labelled set run on every release and on sampled production traffic. Pairwise metrics with the order swap, a judge from a different model family and reported lengths run when a change needs a decision. A release passes when it wins the comparison and clears the thresholds, and when the two patterns disagree on a case, that case gets read by a person. The pattern you pick will still be wrong sometimes, since every LLM judge is, but it will be wrong in a way you planned for rather than one you discover after shipping.
