Why does our LLM-as-judge say everything passes?
A judge from the same model family as the generator tends to rate that family's output favorably — self-preference bias — and a single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise.
Also known as LLM judge bias, self-grading eval
What does this look like in production?
- Nearly every case in the eval suite scores at or near the top of the scale, regardless of known quality differences between them
- A deliberately degraded output (injected typo, wrong fact, missing field) still passes the judge's review
- The judge and the generator are the same model, or from the same provider and generation
- Human reviewers, sampled on the same cases, disagree with the judge meaningfully more often than the team expected
Why does it happen?
LLM-as-judge scoring inherits two separate problems that compound each other. First, models measurably favor output that resembles their own style and reasoning patterns — a judge grading a generator from the same family is, to some degree, grading itself, and tends to be lenient about the exact quirks it would produce too. Second, a rubric that asks only "is this a good response?" without specific, checkable failure criteria gives the judge nothing to anchor a low score on — vague criteria collapse toward uniformly high scores, because there's no defined bar the judge is checking against, only a general impression of plausibility.
The result is a suite that appears to be working (it runs, it returns scores, most things pass) while actually carrying no information: a judge that says yes to almost everything can't distinguish a real regression from noise, which defeats the entire purpose of having it.
How do you confirm it's this?
- Look at the score distribution across your whole eval set — if it clusters tightly near the maximum, the judge isn't discriminating between good and bad cases
- Deliberately inject a known defect into a passing case and confirm the judge catches it — if it still passes, the rubric or judge is too lenient to trust
- Measure agreement between the judge and a small human-labelled holdout (about 50 sampled items is a reasonable start) and treat low agreement as a signal to fix the judge, not the generator
- Check whether the judge model is the same as, or closely related to, the model being graded
How do you fix it?
- Write a rubric with explicit failure definitions
Replace "is this good?" with specific, checkable criteria — what exactly counts as a failure on this dimension — so the judge has something concrete to check against instead of a vibe.
- Use a judge from a different model family than the generator
Grading a model's output with a differently-trained model reduces the self-preference effect that comes from shared training data and stylistic habits.
- Calibrate against human labels
Periodically score a sample with real human review and compare against the judge's scores on the same cases, treating disagreement as a prompt to revise the rubric or judge setup, not as noise to ignore.
- Keep a small human-graded holdout permanently
Retain a fixed set the judge is never trained or tuned against, and re-check agreement on it periodically — a judge that drifts without anyone noticing is worse than no judge at all, since it manufactures false confidence.
What this doesn't cover
This describes leniency and self-preference specifically — an LLM judge can also fail in the opposite direction (overly harsh, or biased against a particular style regardless of quality), which needs the same calibration-against-humans fix but shows up as unexpectedly low scores rather than uniformly high ones.
A Ship Audit runs this full checklist against your actual system and hands back a written, prioritized plan.