How do we know it works, and how do we keep knowing after every change?
Shipping without an eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. means every prompt or model change afterward is a guess about whether quality went up or down. Evaluate is the stage that turns that guess into a number.
What happens at this stage#
A prompt or model change is a code change with no compiler and no test suite, unless a team builds one deliberately. Without a fixed, labelled set of cases, "did this help" gets answered by re-running the one or two examples already open in a tab. Those examples were chosen, consciously or not, because they show the improvement being made. That's an anecdote, not a test, and it will look like a win almost regardless of what happens elsewhere in the real input distribution.
A real eval suite starts from 50–100 cases. They come from actual production traffic and from edge cases that already caused a problem. Invented happy-path examples do not count. Each case has a specific, checkable pass/fail condition, and "sounds good" does not qualify. The suite runs in CI on every prompt, model or retrieval change, the way a unit test suite runs on every code change. A drop in the pass rate blocks the merge like a failing test would. Only a deliberate, reviewed trade-off overrides it.
LLM-as-judgeLLM-as-judge uses another LLM to score a model's outputs against a rubric, at scale. is a genuinely useful tool inside that suite for subjective qualities like tone or faithfulness that a plain string match can't score. But it inherits real biases that have to be actively managed, not assumed away. A judge from the same model family as the generator tends to rate that family's output favorably. A vague rubric ("is this good?") collapses almost everything to a passing score. Calibrate the judge against a small human-labelled holdout periodically. Now and then, inject a known defect into a passing case to confirm the judge still catches it.
Name similarity-is-not-relevance at this stage. A topical eval set misses exactly this failure. Cosine similarity rewards "about the same thing" and ignores the specific identifier, date or negation that decides whether an answer is right. If your eval set is all topical queries, it will pass cleanly right up until real users start asking about specific accounts and order numbers. Build a separate identifier-and-date query set and measure it on its own, not folded into the general average.
Evals and observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. are the before-and-after halves of one discipline. Evals catch a regression before it ships. Observability catches the one that got through anyway. Neither replaces the other. A clean eval run tells you the known risks didn't regress, not that the change is safe in every respect. A trace-first observability setup turns "the AI got worse" from a vague complaint into a specific span and a specific prompt version. It also points to a specific fix.
The common mistake#
The most common mistake at this stage is running the eval suite as an optional script someone can forget, rather than a required CI gate. That means the one time it would have caught something is exactly the time nobody happened to run it.
How do you know this stage is finished?#
- A labelled eval set of 50-100 real cases exists, each with a specific, checkable pass/fail condition.
- The suite is a required CI check on prompt, model, and retrieval changes, not an optional script.
- If an LLM judge is used, it's calibrated against a human-labelled holdout and has been tested against a deliberately injected defect.
- Identifier and date-shaped queries are evaluated as their own set, separate from topical queries.
- Trace-first observability is in place to catch what the eval suite's known cases don't.
What backs this up#
Insights
- Evals Before You Ship: Why AI Features Need Tests Too
The case for treating evals as a required gate, not an optional nice-to-have.
- What Our Coding Agents Did to Get Past the Build Checks
First-hand incidents where a green check hid a wrong page, and the habits that make a check hard to game.
- Your RAG Demo Lied to You
Why a demo passing is weak evidence, and what a real eval set catches that a demo doesn't.
Patterns
- Golden-set gate in CI
The CI mechanics that turn "we have an eval set" into "a regression can't merge."
- Trace-first observability
The after-ship half of this stage's discipline. Evals catch it before; this catches what got through.
Failure modes
- Vibes-based prompt regression
What happens without a labelled set: a change gets graded against whatever examples were already open.
- Judge prefers its own output
The specific bias to calibrate out before trusting an LLM judge's score.
- Similarity is not relevance
The failure a topical-only eval set won't catch. Test identifier and date queries separately.
Glossary
- Eval Suite (Evals)
The artifact this whole stage is built around.
- LLM-as-Judge
One signal in the suite, not the whole suite. See the calibration failure mode above.
- Hallucination
The thing evals exist to catch reliably instead of noticing it by accident.
- Red Teaming
The adversarial counterpart to a golden-set eval. It checks that the system fails safely, not that it gets normal cases right.
- Shadow Deployment
The offline-replay half of the gate: run a candidate against real traffic before any of it is real exposure.
- Acceptance Rate
The same graded-pass-rate discipline this stage runs on a golden set, applied to a human reviewer judging an agent's actual proposed changes.
More
- Quality is a measured number.
The eval methodology this studio actually runs on client engagements, published in full.
- The Eval Stack
The full pillar behind this stage: six eval layers, 24 named checks, five ways an LLM judge lies, and the scoreboard.
- The AI Release Path
The gate surface of this pillar, extended into what happens once a candidate clears it: how it ramps, and how it reverses if it's wrong.
Questions on this stage#
What comes up before and during evaluate.
01 How big does an eval set actually need to be to be useful?
50-100 real cases is a reasonable starting bar. Pull them from actual production traffic and past incidents, not invented ones, each with a specific pass/fail condition rather than a vibe. Small and real beats large and synthetic.
Link to this answer: How big does an eval set actually need to be to be useful?02 Can we just use an LLM to judge our LLM's output?
Yes, but calibrate it. A judge from the same model family as the generator tends to rate that family's output favorably. A vague rubric ("is this good?") collapses almost everything to a passing score. Check agreement against a human-labelled holdout and confirm the judge catches a deliberately injected defect.
Link to this answer: Can we just use an LLM to judge our LLM's output?03 What's the difference between evals and observability?
Evals catch a regression before it ships; observability catches the one that got through anyway. They're the before-and-after halves of the same discipline, and a mature system needs both.
Link to this answer: What's the difference between evals and observability?A Ship Audit checks your actual system against every stage of this handbook and hands back a written, prioritized plan.