Somebody on your team is about to put a number in a slide. The number will be 0.82, it will be labelled "answer quality", and everyone in the room will read it as "82% of our answers are good". That is not what it says.
A judge score is the output of a model that was asked a question about another model's output. It inherits every property of a model answer: it is a sample from a distribution, it moves when you change the prompt, and it is confident whether or not it is right. What makes it useful is not that it is objective. It is that it is consistent, and consistency is what lets you compare Tuesday to Thursday.
The number is an agreement rate with one specific rubric
When a judge returns 0.82 for a run, that is the mean of per-row verdicts produced by a rubric you wrote. Change a word in the rubric and the number moves, without anything about your product changing. This is not a defect to be engineered away, it is the nature of the instrument. A thermometer calibrated in Fahrenheit does not report the same figure as one in Celsius, and neither is lying.
What follows from that is a rule worth writing on the wall:
A judge score is only comparable to another score from the same judge, the same rubric, and the same model.
Every dashboard that violates this rule eventually produces a meeting about a regression that never happened.
Calibrate before you trust, not after you are surprised
Calibration is one afternoon of work and it is the difference between a metric and a decoration.
- Label 50 rows by hand. Not 500. Fifty is enough to find gross disagreement, and small enough that a person will actually do it.
- Run the judge over the same 50. Keep the raw verdicts, not the mean.
- Build the confusion matrix. Where does the judge pass what you failed, and fail what you passed?
- Read the ten worst disagreements. Every time we have done this, the rubric was ambiguous rather than the judge being stupid.
- Fix the rubric and repeat once. Two rounds is usually enough.
The output of this is not a better score. It is a sentence you can say out loud: "when this judge says fail, it agrees with a human nine times in ten." That sentence is what makes the number usable in a release decision.
Thresholds are a product decision, not a statistical one
Teams spend a long time arguing about whether the threshold should be 0.7 or 0.75, as though there were a correct answer hiding in the data. There is not. The threshold encodes how much you would rather ship a bad answer than block a good release, and that trade is owned by whoever answers for the product, not by whoever wrote the metric.
What the data can tell you is the cost of each choice. Sort your calibration rows by judge score and read the band around your candidate threshold. If moving from 0.70 to 0.75 turns four passes into failures and three of those four really were bad, you have learned something. If it turns forty into failures, your judge is not discriminating in that range and no threshold there will hold.
Three failure modes worth knowing by name
Position bias. Ask a judge to pick the better of two answers and it prefers the first one more often than chance. Always score both orders and average, or use a single-answer rubric.
Verbosity bias. Longer answers score higher, roughly regardless of content. If your change made answers longer, expect a lift that is not real. Check length alongside quality, always.
Self-preference. A judge from the same family as the system under test scores it more kindly. Use a different provider for the judge where you can, and at minimum know which way the bias points.
What to do on Monday
Pick the one metric that is currently in a slide. Label fifty rows against it. You will find out in an afternoon whether the number in that slide means what the room thinks it means, and that is a cheaper way to find out than the alternative.
