# Why does our LLM-as-judge say everything passes? Source: https://customlabs.io/failure-modes/judge-prefers-its-own-output/ Updated: 2026-09-17 Evals & quality # Why does our LLM-as-judge say everything passes? Updated September 17, 2026 · First published July 26, 2026 · 4 min read · Answer A judge from the same model family as the generator tends to rate that family's output favorably. This is self-preference bias. A single vague rubric ('is this good?') collapses almost everything to a passing score, so the [eval suite](https://customlabs.io/glossary/eval-suite/) stops being able to tell a real regression from noise. Also known as LLM judge bias, self-grading eval Symptoms ## What does this look like in production? - Nearly every case in the eval suite scores at or near the top of the scale, regardless of known quality differences between them - A deliberately degraded output (injected typo, wrong fact, missing field) still passes the judge's review - The judge and the generator are the same model, or from the same provider and generation - Human reviewers, sampled on the same cases, disagree with the judge meaningfully more often than the team expected Root cause ## Why does it happen? [LLM-as-judge](https://customlabs.io/glossary/llm-as-judge/) scoring inherits two separate problems that compound each other. First, models measurably favor output that resembles their own style and reasoning patterns. A judge grading a generator from the same family is, to some degree, grading itself, and tends to be lenient about the exact quirks it would produce too. Second, a rubric that asks only "is this a good response?" without specific, checkable failure criteria gives the judge nothing to anchor a low score on. Vague criteria collapse toward uniformly high scores, because there's no defined bar the judge is checking against, only a general impression of plausibility. The result is a suite that appears to be working (it runs, it returns scores, most things pass) while actually carrying no information: a judge that says yes to almost everything can't distinguish a real regression from noise, which defeats the entire purpose of having it. Detect ## How do you confirm it's this? - Look at the score distribution across your whole eval set. If it clusters tightly near the maximum, the judge isn't discriminating between good and bad cases - Deliberately inject a known defect into a passing case and confirm the judge catches it. If it still passes, the rubric or judge is too lenient to trust - Measure agreement between the judge and a small human-labelled holdout (about 50 sampled items is a reasonable start) and treat low agreement as a signal to fix the judge, not the generator - Check whether the judge model is the same as, or closely related to, the model being graded Fix ## How do you fix it? - Write a rubric with explicit failure definitions Replace "is this good?" with specific, checkable criteria that define what exactly counts as a failure on this dimension, so the judge has something concrete to check against instead of a vibe. - Use a judge from a different model family than the generator Grading a model's output with a differently-trained model reduces the self-preference effect that comes from shared training data and stylistic habits. - Calibrate against human labels Periodically score a sample with real human review and compare against the judge's scores on the same cases, treating disagreement as a prompt to revise the rubric or judge setup, not as noise to ignore. - Keep a small human-graded holdout permanently Retain a fixed set the judge is never trained or tuned against, and re-check agreement on it periodically. A judge that drifts without anyone noticing is worse than no judge at all, since it manufactures false confidence. Limits ## What this doesn't cover This describes leniency and self-preference specifically. An LLM judge can also fail in the opposite direction (overly harsh, or biased against a particular style regardless of quality). That needs the same calibration-against-humans fix but shows up as unexpectedly low scores rather than uniformly high ones. ### Sources - [Stanford CRFM - Holistic Evaluation of Language Models (HELM)](https://crfm.stanford.edu/helm/) The benchmark suite our check bank draws its evaluation method from. Retrieved 2026-08-24. - [NIST - Generative AI Profile (NIST AI 600-1)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) The generative-AI extension to the framework our eval and security controls reference. Retrieved 2026-08-24. In the Handbook [04 Evaluate](https://customlabs.io/handbook/evaluate/) Prevented by [Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/) Topics [Evals & Observability](https://customlabs.io/topics/evals-observability/) Related reading [Glossary: LLM-as-Judge](https://customlabs.io/glossary/llm-as-judge/)[Insight: Evals before you ship](https://customlabs.io/insights/evals-before-you-ship/) Not sure if this is the one? A Ship Audit runs this full checklist against your actual system and hands back a written, prioritized plan. [Book a Ship Audit →](https://customlabs.io/diagnostic/ship-audit/)