How do we know it works, and how do we keep knowing after every change?
Shipping without an eval suite means every prompt or model change afterward is a guess about whether quality went up or down — evaluate is the stage that turns that guess into a number.
What actually happens at this stage
A prompt or model change is a code change with no compiler and no test suite, unless a team builds one deliberately. Without a fixed, labelled set of cases and a checkable expected property for each, "did this help" gets answered by re-running the one or two examples already open in a tab — chosen, consciously or not, because they demonstrate the improvement being made. That's an anecdote, not a test, and it will look like a win almost regardless of what happens elsewhere in the real input distribution.
A real eval suite starts from 50–100 cases pulled from actual production traffic and edge cases that already caused a problem, not invented happy-path examples, each with a specific, checkable pass/fail condition — not "sounds good." It runs in CI on every prompt, model, or retrieval change, the same way a unit test suite runs on every code change, and a drop in the pass rate blocks the merge the same way a failing test would, unless the regression is a deliberate, reviewed trade-off.
LLM-as-judge is a genuinely useful tool inside that suite for subjective qualities like tone or faithfulness that a plain string match can't score — but it inherits real biases that have to be actively managed, not assumed away. A judge from the same model family as the generator tends to rate that family's output favorably, and a vague rubric ("is this good?") collapses almost everything to a passing score. Calibrate the judge against a small human-labelled holdout periodically, and deliberately inject a known defect into a passing case now and then to confirm the judge still catches it.
Similarity-is-not-relevance is worth naming explicitly at this stage because it's exactly the kind of failure a topical eval set misses: cosine similarity rewards "about the same thing," not correctness on the specific identifier, date, or negation that actually decides whether an answer is right. If your eval set is all topical queries, it will pass cleanly right up until real users start asking about specific accounts and order numbers — build a separate identifier-and-date query set and measure it on its own, not folded into the general average.
Evals and observability are the before-and-after halves of the same discipline: evals catch a regression before it ships, observability catches the one that got through anyway. Neither replaces the other — a clean eval run tells you the known risks didn't regress, not that the change is safe in every respect — and a trace-first observability setup is what turns "the AI got worse" from a vague complaint into a specific span, a specific prompt version, and a specific fix.
The common mistake
The most common mistake at this stage is running the eval suite as an optional script someone can forget, rather than a required CI gate — which means the one time it would have caught something is exactly the time nobody happened to run it.
How do you know this stage is finished?
- A labelled eval set of 50-100 real cases exists, each with a specific, checkable pass/fail condition.
- The suite is a required CI check on prompt, model, and retrieval changes, not an optional script.
- If an LLM judge is used, it's calibrated against a human-labelled holdout and has been tested against a deliberately injected defect.
- Identifier and date-shaped queries are evaluated as their own set, separate from topical queries.
- Trace-first observability is in place to catch what the eval suite's known cases don't.
What backs this up
Insights
- Evals Before You Ship: Why AI Features Need Tests Too
The case for treating evals as a required gate, not an optional nice-to-have.
- Your RAG Demo Lied to You
Why a demo passing is weak evidence — and what a real eval set catches that a demo doesn't.
Patterns
- Golden-set gate in CI
The CI mechanics that turn "we have an eval set" into "a regression can't merge."
- Trace-first observability
The after-ship half of this stage's discipline — evals catch it before, this catches what got through.
Failure modes
- Vibes-based prompt regression
What happens without a labelled set: a change gets graded against whatever examples were already open.
- Judge prefers its own output
The specific bias to calibrate out before trusting an LLM judge's score.
- Similarity is not relevance
The failure a topical-only eval set won't catch — test identifier and date queries separately.
Glossary
- Eval Suite (Evals)
The artifact this whole stage is built around.
- LLM-as-Judge
One signal in the suite, not the whole suite — see the calibration failure mode above.
- Hallucination
The thing evals exist to catch reliably instead of noticing it by accident.
- Red Teaming
The adversarial counterpart to a golden-set eval — checks that the system fails safely, not that it gets normal cases right.
More
- Quality is a number, not a vibe.
The eval methodology this studio actually runs on client engagements, published in full.
- The Eval Stack
The full pillar behind this stage: six eval layers, 24 named checks, five ways an LLM judge lies, and the scoreboard.
Questions on this stage
What comes up before and during evaluate.
01 How big does an eval set actually need to be to be useful?
50-100 real cases is a reasonable starting bar — pulled from actual production traffic and past incidents, not invented, each with a specific pass/fail condition rather than a vibe. Small and real beats large and synthetic.
02 Can we just use an LLM to judge our LLM's output?
Yes, but calibrate it. A judge from the same model family tends to favor that family's output, and a vague rubric collapses to near-uniform passing scores — check agreement against a human-labelled holdout and confirm the judge catches a deliberately injected defect.
03 What's the difference between evals and observability?
Evals catch a regression before it ships; observability catches the one that got through anyway. They're the before-and-after halves of the same discipline, and a mature system needs both.
A Ship Audit checks your actual system against every stage of this handbook and hands back a written, prioritized plan.