Contract checks
Catches: Malformed output: broken JSON, a missing field, or a tool call with the wrong argument shape.
Blind to: Whether a well-formed answer is actually correct.
Runs when: Every request, inline, before the response is used
Evals
The Eval Stack: how to know an AI system actually works.
Six layers, 24 named checks, five ways an LLM judge lies, and the numbers worth putting on a dashboard. This is the discipline behind the evaluate stage of the Handbook, not a pitch for a tool.
A demo proves a system can work once, on the cases you picked to show it off. An eval proves it works on the cases you don't get to pick: the ones a real user actually sends, including the ones that already broke it before. Most teams ship on vibes, discover the regression in production, and then can't say whether the fix helped, because nothing was measuring in the first place.
Evaluation isn't one gate. It's a stack of layers, and each one catches a class of failure the layer below it is blind to. Skipping a layer doesn't lower your score. It means you stop measuring that class of failure entirely, and find out about it in production instead.
The stack
Six layers, stacked, not gated.#
Each layer catches something the one below it can't see. None of them replace each other.
Golden-set task evals
Catches: A prompt, model, or retrieval change that quietly makes real cases worse.
Blind to: Retrieval quality in isolation, and any case shape the set doesn't include.
Runs when: Every change to a prompt, model, or retrieval config, in CI
Retrieval evals
Catches: The retriever handing the generator the wrong context, or none at all.
Blind to: How well the generator uses good context once it actually has it.
Runs when: Alongside golden-set evals, scored as its own number
Trajectory evals
Catches: An agent that loops, calls the wrong tool, or never terminates.
Blind to: Whether the final answer, once it stops, was any good.
Runs when: Pre-release, against scripted multi-step scenarios
Online signals
Catches: Everything the offline suite never saw, because this is real traffic.
Blind to: Anything that fails silently and never gets reported or escalated.
Runs when: Continuous, on live production traffic
Adversarial & safety
Catches: Prompt injection, jailbreaks, PII leakage, and tool misuse, under deliberate attack.
Blind to: Ordinary quality; it only answers whether the system can be made to misbehave on purpose.
Runs when: Pre-release, and after any change to tools or the system prompt
The check bank
24 checks, six layers.#
Filter by when it runs (pre-merge, pre-release, continuous), then copy the visible list as a Markdown checklist.
Showing all 24 checks
01 Contract checks
The cheapest layer, and the one most teams already half-have: does the response even parse. A schema check can't tell you the answer is right. It only tells you the answer is shaped the way the next piece of code expects. A wrongly-shaped response breaks that next step regardless of whether the content was good, so this check runs on every single request, in production and in CI alike.
- Measures
- Schema-pass rate, as a %
- Build it
- A JSON Schema or typed parser on every structured output; reject and surface a distinguishable error, not a silent retry loop.
- Catches
- A malformed response breaking the next step in the chain.
- Evidence
- The schema definitions plus the validation-failure rate, tracked separately from execution failures.
- Measures
- Invalid tool-call rate, as a %
- Build it
- Enums and patterns enforced server-side; a mutating call has to trace back to a value an earlier read actually returned, never one the model invented.
- Catches
- Tool argument hallucination: a plausible-looking ID the model made up because it couldn't recall the real one.
- Evidence
- The tool schema definitions plus the validation-failure log.
- Measures
- Silent-failure rate, as a %: tool calls that returned success with nothing usable inside
- Build it
- A typed status enum (ok / empty / error) instead of HTTP 200 with a failure buried in the response body.
- Catches
- Silent tool failure: the model reports an absence of data as a fact because nothing told it otherwise.
- Evidence
- The tool response schema, plus a sample of an error actually surfaced as status: error, not a 200.
Track what share of production traffic actually exercises the schema check, beyond what CI covers.
Continuous- Measures
- Contract-check coverage, as a % of live requests
- Build it
- Log a validation pass/fail on every production request, including request shapes no test author thought to write.
- Catches
- A schema gap on a request shape nobody wrote a test case for.
- Evidence
- The coverage dashboard, broken out by request type.
02 Golden-set task evals
The layer most people mean when they say "we have evals." It is a fixed, labelled set of real cases. Each case carries a checkable pass/fail condition, run as a required CI gate. It answers "did this change help or hurt" with a number, not a re-run of whichever two examples happened to be open in a tab. It says nothing about a case shape the set never included. That is exactly why the next four layers exist.
Seed the golden set from production traces, not from the prompts you wrote while building.
Pre-merge- Measures
- Set size, as a count of labelled cases (50–100 to start)
- Build it
- Pull real traces plus every past incident case; label each with a specific pass/fail condition, not a description of the desired vibe.
- Catches
- An eval suite that only ever re-tests the happy path its own author already knew would pass.
- Evidence
- The labelled case set, with provenance tagged per case: production trace vs. authored.
- Measures
- Held-out pass rate vs. visible-set pass rate, as two separate percentages
- Build it
- Split the set at creation time; only a reviewer, never the person iterating on the prompt, can see the held-out half.
- Catches
- A prompt tuned against the same cases used to grade it: the eval-suite version of training on the test set.
- Evidence
- The set-split record and both pass rates reported side by side, never blended into one number.
Run the golden set on every prompt, model, or retrieval change, and fail the merge on a regression.
Pre-merge- Measures
- Pass rate, as a %, tracked per commit
- Build it
- A required CI check, not an optional script. It gets the same gate a unit-test suite would, applied to a prompt change.
- Catches
- A regression that ships because nobody happened to re-run the eval by hand that day.
- Evidence
- The CI configuration plus the pass-rate history across recent merges.
- Measures
- Edge-case share of the set, as a %
- Build it
- Tag each case by origin at label time: routine, edge case, or past incident. Report the mix alongside the raw count.
- Catches
- A set that looks comprehensive by count but is quietly all easy cases.
- Evidence
- The case-origin breakdown, reported next to the pass rate it produced.
03 Retrieval evals
Retrieval and generation fail independently, so they need separate scores. Otherwise a regression in one hides behind a passing average from the other. Recall@k, precision, and groundedness are measured against the retrieved chunk itself, before the generator ever gets a turn. That is the only way to tell "the model wrote a bad answer" apart from "the model wrote a good answer to the wrong context."
- Measures
- Recall@k, as a %
- Build it
- Label the correct source chunk per query, then check whether it's in the top-k retrieved set before the generator ever sees it.
- Catches
- A generator that sounds confident on context that never contained the answer in the first place.
- Evidence
- The recall@k report, scored independently of the generation eval it feeds.
Evaluate identifier and date-shaped queries as their own set, separate from topical queries.
Pre-merge- Measures
- Identifier-query pass rate, as a %, reported apart from the general average
- Build it
- Build a query set of account numbers, order IDs, and dates specifically. Cosine similarity rewards topical resemblance, not an exact identifier match.
- Catches
- A retrieval eval that passes cleanly on topical queries and fails the moment a real user asks about a specific account.
- Evidence
- The identifier-query set and its pass rate, reported as its own line, never folded into the topical average.
- Measures
- Groundedness rate, as a %
- Build it
- An LLM-judge or NLI-style check comparing each claim in the answer to the specific source span it should trace back to.
- Catches
- A hallucinated answer dressed up around real-looking retrieved context.
- Evidence
- The groundedness scorer output, alongside the source span it checked each claim against.
- Measures
- Index-staleness lag, as the time between a source change and the index reflecting it
- Build it
- Change-data-capture ingest instead of a timer-based recrawl, with a tombstone record for anything deleted at the source.
- Catches
- The index serving a document that was deleted from the source months ago.
- Evidence
- The freshness-lag metric, plus a sample tombstone applied against a real deletion.
04 Trajectory evals
Once a system takes more than one step, the run itself becomes something to grade, separately from the answer at the end of it. Did it terminate? How many steps did it take, which tools did it call, at what cost? A trajectory eval catches the run that technically reached a correct answer by way of three unnecessary tool calls and a near-infinite retry. That defect is invisible to the golden-set layer above, because it only grades the final output.
- Measures
- Terminal-state rate, as a % split across success / failure / escalated
- Build it
- A forced terminal state, enforced by the harness on a step, token, and wall-clock budget. The model never decides this for itself.
- Catches
- An agent that runs for 40 minutes and produces nothing.
- Evidence
- The terminal-state distribution across a sample of recent runs.
Score whether the agent called the right tool for a scripted scenario, beyond whether it finished.
Pre-release- Measures
- Tool-choice accuracy, as a %
- Build it
- A labelled set of scripted multi-step scenarios with the expected tool sequence, graded against the sequence the agent actually called.
- Catches
- An agent that reaches a plausible answer by way of the wrong tool, or skips one the task actually needed.
- Evidence
- The scenario set with expected vs. actual tool sequences side by side.
- Measures
- Steps per completed task, as a distribution: median and p95
- Build it
- Log step count per run and compare it against the ceiling set in the run's bounded-loop budget.
- Catches
- A retry loop hiding inside a run that technically completed.
- Evidence
- The step-count distribution, reported alongside the configured ceiling.
Report cost per completed task, not per call.
Pre-release- Measures
- $ per completed task
- Build it
- Divide total spend for the scenario set by the count of runs that reached a genuine terminal success, not by the number of API calls made.
- Catches
- A retry loop hiding inside a good-looking per-call average.
- Evidence
- The cost-per-task figure, reported alongside the calls-to-completions ratio.
05 Online signals
No offline set, however carefully built, covers every input production will actually throw at the system. This is the layer that measures what happens once it does. It tracks task completion, escalation to a human, and how much a person had to edit an accepted answer. It also tracks cost, latency, and whether any of those numbers drift over time. This is production truth, not a proxy for it.
- Measures
- Task completion rate, as a %
- Build it
- Instrument the production flow for a defined 'completed' event, tracked separately from the offline eval's pass/fail condition.
- Catches
- A system that passes every offline eval and still fails in ways the eval set never anticipated.
- Evidence
- The production completion-rate dashboard, trended over time.
- Measures
- Escalation rate, as a %, broken out by step
- Build it
- Log every handoff event with the specific step it occurred at, beyond a running total.
- Catches
- A quality regression that shows up as users bailing out to a human, before it ever shows up as a lower eval score.
- Evidence
- The escalation-rate trend, segmented by the step in the flow it fires from.
Measure edit distance between the model's draft and what a human actually accepted or sent.
Continuous- Measures
- Edit distance, as a normalized % change
- Build it
- Capture the final accepted version wherever a human can edit the model's draft before it ships, and diff it against the original.
- Catches
- A model whose answers 'pass' the eval suite but still need a heavy rewrite before anyone will use them.
- Evidence
- The edit-distance trend over time.
- Measures
- Drift, as the delta between the last CI pass rate and a live-traffic re-score
- Build it
- Periodically re-run the golden-set judge against sampled live traffic and diff the result against the CI number.
- Catches
- A provider-side model upgrade that silently changes quality between merges.
- Evidence
- The drift report comparing the CI score against the live re-score.
06 Adversarial & safety
A system can pass every quality check above and still be talked into something it shouldn't do, because none of those checks were trying to attack it. This layer runs the labelled, adversarial equivalent of the golden set: injected instructions, known jailbreak patterns, PII-extraction attempts. It runs against the same tool-enabled path a real user reaches, and answers a narrower question than the rest of the stack on purpose.
- Measures
- Injection-resistance rate, as a %
- Build it
- Plant adversarial instructions inside retrieved content and tool outputs, beyond the user-facing prompt.
- Catches
- A document in your own knowledge base that reads like a command, and gets treated like one.
- Evidence
- The injection test set and its pass rate, tracked in CI alongside the quality suite.
- Measures
- Jailbreak-resistance rate, as a %
- Build it
- Maintain a running set of public jailbreak techniques plus any that have worked against this system before, and re-test on every guardrail or system-prompt change.
- Catches
- A guardrail that quietly stopped working after a prompt edit nobody re-tested it against.
- Evidence
- The jailbreak test set and its pass rate, versioned against the system-prompt revision it ran on.
- Measures
- PII-leakage rate, as a %
- Build it
- A labelled set of prompts designed to extract PII from context the system has access to, checked against what actually comes back.
- Catches
- A retrieval or memory feature handing back personal data outside its intended scope.
- Evidence
- The leakage test set and its pass rate, plus the redaction rule each case is checked against.
- Measures
- Tool-abuse block rate, as a %
- Build it
- Adversarial scenarios targeting every tool that writes, sends, or deletes, confirming the human checkpoint fires or the call is rejected outright.
- Catches
- An agent talked into issuing a refund or sending a message it shouldn't have.
- Evidence
- The scenario set, plus a trace showing the checkpoint firing on a live example.
No checks match that combination. Clear a filter to see more.
Five ways a judge lies
LLM-as-judge is a tool, not a verdict.#
Every one of these produces a confidently wrong number, not an obviously broken one.
Self-preference
Looks like: A judge from the same model family as the generator rates that family's output a little higher, across the board, for no stated reason.
Guardrail: Use a judge from a different model family than the generator, or budget for periodic cross-family spot checks.
Position bias
Looks like: In a side-by-side comparison, the judge favors whichever answer it saw first, or second, regardless of content.
Guardrail: Randomize answer order per comparison and average both orderings before trusting a verdict.
Verbosity bias
Looks like: The longer answer wins even when it says the same thing at greater length.
Guardrail: Score against a rubric that names length as a non-factor explicitly, and spot-check whether shorter correct answers are losing to longer ones.
Rubric drift
Looks like: The same rubric, applied a month apart, grades a static test case differently. The judge model or its default settings changed underneath you.
Guardrail: Pin the judge model and version; treat a judge upgrade as a breaking change that needs re-baselining, not a free improvement.
No human-calibration baseline
Looks like: The judge's pass rate looks stable, and nobody has checked it against a human rater in months.
Guardrail: Re-run a human-labelled holdout against the judge periodically and confirm agreement, beyond internal consistency.
The scoreboard
Six numbers, and how each one lies.#
Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.
Golden-set pass rate
- Why it matters
- Says whether the cases you already know matter still work after a change.
- How it misleads
- A rising rate against a small or unchanged set can mean the product improved, or it can mean the set stopped being hard enough to catch anything.
- Pair with
- Set coverage
Set coverage
- Why it matters
- How much of the real input distribution the golden set actually represents, including the edge cases that already caused an incident.
- How it misleads
- A large case count says nothing if the set is all easy, topical cases; a small set of hard-won incident cases can be worth more.
- Pair with
- Golden-set pass rate
Recall@k
- Why it matters
- Whether the retriever hands the generator the right context at all, measured before generation quality can hide the gap.
- How it misleads
- A recall number can look fine on topical queries and still miss badly on identifier or date-shaped ones, if those are not scored separately.
- Pair with
- Groundedness rate
Groundedness rate
- Why it matters
- Whether the generated answer is actually traceable to the retrieved context, beyond being fluent and topically plausible.
- How it misleads
- A high groundedness score paired with low recall just means the model is faithfully repeating context that was wrong to begin with.
- Pair with
- Recall@k
p95 latency
- Why it matters
- The tail a user actually notices. A good median can hide a slow tail that drives complaints and abandonment.
- How it misleads
- Watching only the median lets a growing p95 hide in an average that still looks fine.
- Pair with
- Cost per completed task
Cost per completed task
- Why it matters
- What a feature actually costs to run per real outcome, not per API call.
- How it misleads
- Cost per call undercounts it. A retry loop, or a fallback to a pricier model on failure, vanishes entirely from a per-call number.
- Pair with
- p95 latency
What this is built from
Verifiable, not claimed.#
No invented benchmark numbers, no client names. Just what's already documented on this site.
- Golden-set gate in CI and trace-first observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. are patterns this site already documents in full. This page connects them into one stack instead of treating either as a standalone technique. Pattern: Golden-set gate in CI
- Judge-prefers-its-own-output, similarity-is-not-relevance, silent-tool-failure, and vibes-based-prompt-regression are named failure modes on this site, not hypotheticals. Each is the reason a specific layer above exists. Failure mode: Judge prefers its own output
- CodeHerder runs a golden-set gate in CI on every task branch before it merges. That is layer 2 of this stack, applied across an entire agent fleet instead of one feature. CodeHerder
- CostMon is being built to answer the layer-4 and scoreboard question, cost per completed task, as a self-serve number instead of a monthly reconciliation project. CostMon (coming soon)
- The eval methodology on Capabilities is the short version of this argument; this page is the full stack behind it. Capabilities: Evals methodology
Sources
- Stanford CRFM - Holistic Evaluation of Language Models (HELM)
The benchmark suite our check bank draws its evaluation method from. Retrieved 2026-08-24.
- NIST - AI Risk Management Framework (AI RMF 1.0)
The risk-management functions our governance and adoption controls map onto. Retrieved 2026-08-24.
- NIST - Generative AI Profile (NIST AI 600-1)
The generative-AI extension to the framework our eval and security controls reference. Retrieved 2026-08-24.
Not sure which layer you're missing
A Ship Audit runs this stack against your specific system and tells you which layers are live risks, not just which ones are theoretically incomplete.
Questions
Before you build the suite.#
What teams ask us before they scope an eval stack.
01 How big does a golden set actually need to be?
50–100 real cases is a reasonable starting bar, pulled from production traffic and past incidents rather than invented. See the golden-set layer above. A held-out slice the prompt author never sees matters more than raw count.
Link to this answer: How big does a golden set actually need to be?02 Do we need all six layers on day one?
No. Start with contract checks and a golden-set gate in CI. Both are cheap and catch the two failure modes that do the most damage. Add trajectory and adversarial evalsAn eval suite is a versioned set of test cases that measures whether outputs are good. once there's an agent and real tool access worth protecting.
Link to this answer: Do we need all six layers on day one?03 Can we just use an LLM to judge our own output?
Yes, with guardrails: pin the judge model and version, randomize comparison order, and calibrate against a human-labelled holdout periodically. See the five ways a judge lies above; each one is a way a judge score can look fine while being quietly wrong.
Link to this answer: Can we just use an LLM to judge our own output?04 What's the difference between an eval and observability?
Evals catch a regression before it ships. Observability, the online-signals layer, catches the one that got through anyway. Neither replaces the other.
Link to this answer: What's the difference between an eval and observability?05 Isn't this a lot of infrastructure for a small feature?
Scope it to what the feature can do. A read-only summarizer barely needs the retrieval and trajectory layers; something that can send messages or issue refunds needs the full stack, adversarial checks included.
Link to this answer: Isn't this a lot of infrastructure for a small feature?06 How is this different from the eval methodology on Capabilities?
That page is the three-step summary. This is the full stack behind it: six layers, 24 named checks, the ways a judge lies, and the numbers worth putting on a dashboard.
Link to this answer: How is this different from the eval methodology on Capabilities?