Why does our LLM-as-judge say everything passes?

What does this look like in production?

• Nearly every case in the eval suite scores at or near the top of the scale, regardless of known quality differences between them
• A deliberately degraded output (injected typo, wrong fact, missing field) still passes the judge's review
• The judge and the generator are the same model, or from the same provider and generation
• Human reviewers, sampled on the same cases, disagree with the judge meaningfully more often than the team expected

Why does it happen?

LLM-as-judgeLLM-as-judge uses another LLM to score a model's outputs against a rubric, at scale. scoring inherits two separate problems that compound each other. First, models measurably favor output that resembles their own style and reasoning patterns. A judge from the same family as the generator is, to some degree, grading itself. It tends to be lenient about the exact quirks it would produce too. Second, a rubric that asks only "is this a good response?" gives the judge nothing to anchor a low score on. It has no specific, checkable failure criteria. Vague criteria collapse toward uniformly high scores, because there's no defined bar the judge is checking against, only a general impression of plausibility.

The result is a suite that appears to work. It runs, it returns scores, most things pass. Yet it carries no information. A judge that says yes to almost everything cannot tell a real regression from noise. That defeats the purpose of having it.

How do you confirm it's this?

• Look at the score distribution across your whole eval set. If it clusters tightly near the maximum, the judge isn't discriminating between good and bad cases
• Deliberately inject a known defect into a passing case and confirm the judge catches it. If it still passes, the rubric or judge is too lenient to trust
• Measure agreement between the judge and a small human-labelled holdout. About 50 sampled items is a reasonable start. Treat low agreement as a sign that the judge needs fixing
• Check whether the judge model is the same as, or closely related to, the model being graded

How do you fix it?

1. Write a rubric with explicit failure definitions

   Replace "is this good?" with specific, checkable criteria that define what counts as a failure on this dimension. The judge then has something concrete to check against.

2. Use a judge from a different model family than the generator

   Grading a model's output with a differently-trained model reduces the self-preference effect that comes from shared training data and stylistic habits.

3. Calibrate against human labels

   Periodically score a sample with real human review. Compare it against the judge scores on the same cases. Treat disagreement as a prompt to revise the rubric or judge setup.

4. Keep a small human-graded holdout permanently

   Retain a fixed set the judge is never trained or tuned against, and re-check agreement on it periodically. A judge that drifts without anyone noticing is worse than no judge at all, since it manufactures false confidence.

What this doesn't cover

This describes leniency and self-preference specifically. An LLM judge can also fail in the opposite direction (overly harsh, or biased against a particular style regardless of quality). That needs the same calibration-against-humans fix but shows up as unexpectedly low scores rather than uniformly high ones.