Contract checks
Catches: Malformed output: broken JSON, a missing field, or a tool call with the wrong argument shape.
Blind to: Whether a well-formed answer is correct.
Runs when: Every request, inline, before the response is used
Evals
The Eval Stack: how to know an AI system works.
Six layers, 24 named checks, five ways an LLM judge lies, and the numbers worth putting on a dashboard. This is the discipline behind the evaluate stage of the Handbook. It is not a pitch for a tool.
A demo proves a system can work once, on the cases you picked to show it off. An eval proves it works on the cases you do not pick. A real user sends those cases, and they include cases that broke the system before. Most teams ship on vibes and find the regression in production. They cannot say whether the fix helped, because nothing measured the system.
Evaluation isn't one gate. It's a stack of layers, and each one catches a class of failure the layer below it is blind to. Skipping a layer doesn't lower your score. It means you stop measuring that class of failure entirely, and find out about it in production instead.
The stack
Six layers, stacked.#
Each layer catches something the one below it can't see. None of them replace each other.
Golden-set task evals
Catches: A prompt, model, or retrieval change that makes real cases worse without notice.
Blind to: Retrieval quality in isolation, and any case shape the set doesn't include.
Runs when: Every change to a prompt, model, or retrieval config, in CI
Retrieval evals
Catches: The retriever handing the generator the wrong context, or none at all.
Blind to: How well the generator uses good context once it has it.
Runs when: Alongside golden-set evals, scored as its own number
Trajectory evals
Catches: An agent that loops, calls the wrong tool, or never terminates.
Blind to: Whether the final answer, once it stops, was any good.
Runs when: Pre-release, against scripted multi-step scenarios
Online signals
Catches: Everything the offline suite never saw, because this is real traffic.
Blind to: Anything that fails silently and never gets reported or escalated.
Runs when: Continuous, on live production traffic
Adversarial & safety
Catches: Prompt injection, jailbreaks, PII leakage, and tool misuse, under deliberate attack.
Blind to: Ordinary quality; it only answers whether the system can be made to misbehave on purpose.
Runs when: Pre-release, and after any change to tools or the system prompt
The check bank
24 checks, six layers.#
Filter by when it runs (pre-merge, pre-release, continuous), then copy the visible list as a Markdown checklist.
Showing all 24 checks
01 Contract checks
The cheapest layer, and the one most teams already half-have: does the response even parse. A schema check can't tell you the answer is right. It only tells you the answer is shaped the way the next piece of code expects. A wrongly-shaped response breaks that next step, even when the content was good. This check runs on every request, in production and in CI.
- Measures
- Schema-pass rate, as a %
- Build it
- A JSON Schema or typed parser on every structured output; reject the response and show a distinct error. Do not use a silent retry loop.
- Catches
- A malformed response breaking the next step in the chain.
- Evidence
- The schema definitions plus the validation-failure rate, tracked separately from execution failures.
Validate every tool-call argument against a typed contract. A free-form string is not enough.
Continuous- Measures
- Invalid tool-call rate, as a %
- Build it
- Enums and patterns enforced server-side; a mutating call must trace back to a value that an earlier read returned. The model must not invent that value.
- Catches
- Tool argument hallucination: a plausible-looking ID the model made up because it couldn't recall the real one.
- Evidence
- The tool schema definitions plus the validation-failure log.
- Measures
- Silent-failure rate, as a %: tool calls that returned success with nothing usable inside
- Build it
- A typed status enum (ok / empty / error) instead of HTTP 200 with a failure buried in the response body.
- Catches
- Silent tool failure: the model reports an absence of data as a fact because nothing told it otherwise.
- Evidence
- The tool response schema, plus a sample of an error that the tool surfaced as status: error with no 200.
Track what share of production traffic exercises the schema check, beyond what CI covers.
Continuous- Measures
- Contract-check coverage, as a % of live requests
- Build it
- Log a validation pass/fail on every production request, including request shapes no test author thought to write.
- Catches
- A schema gap on a request shape nobody wrote a test case for.
- Evidence
- The coverage dashboard, broken out by request type.
02 Golden-set task evals
The layer most people mean when they say "we have evals." It is a fixed, labelled set of real cases. Each case carries a checkable pass/fail condition, run as a required CI gate. It answers "did this change help or hurt" with a number. A re-run of two examples from an open tab cannot do that. It says nothing about a case shape the set never included. The next four layers exist for that reason.
- Measures
- Set size, as a count of labelled cases (50–100 to start)
- Build it
- Pull real traces plus every past incident case; label each with a specific pass/fail condition. Do not label with a description of the desired vibe.
- Catches
- An eval suite that only ever re-tests the happy path its own author already knew would pass.
- Evidence
- The labelled case set, with provenance tagged per case: production trace vs. authored.
- Measures
- Held-out pass rate vs. visible-set pass rate, as two separate percentages
- Build it
- Split the set at creation time; only a reviewer, never the person iterating on the prompt, can see the held-out half.
- Catches
- A prompt tuned against the same cases used to grade it: the eval-suite version of training on the test set.
- Evidence
- The set-split record and both pass rates reported side by side, never blended into one number.
Run the golden set on every prompt, model, or retrieval change, and fail the merge on a regression.
Pre-merge- Measures
- Pass rate, as a %, tracked per commit
- Build it
- A required CI check. It gets the same gate a unit-test suite would, applied to a prompt change.
- Catches
- A regression that ships because nobody happened to re-run the eval by hand that day.
- Evidence
- The CI configuration plus the pass-rate history across recent merges.
- Measures
- Edge-case share of the set, as a %
- Build it
- Tag each case by origin at label time: routine, edge case, or past incident. Report the mix alongside the raw count.
- Catches
- A set that looks comprehensive by count but is all easy cases.
- Evidence
- The case-origin breakdown, reported next to the pass rate it produced.
03 Retrieval evals
Retrieval and generation fail independently, so they need separate scores. Otherwise a regression in one hides behind a passing average from the other. Recall@k, precision, and groundedness are measured against the retrieved chunk itself, before the generator ever gets a turn. Only this tells "a bad answer" apart from "a good answer to the wrong context."
- Measures
- Recall@k, as a %
- Build it
- Label the correct source chunk per query, then check whether it's in the top-k retrieved set before the generator ever sees it.
- Catches
- A generator that sounds confident on context that never contained the answer in the first place.
- Evidence
- The recall@k report, scored independently of the generation eval it feeds.
Evaluate identifier and date-shaped queries as their own set, separate from topical queries.
Pre-merge- Measures
- Identifier-query pass rate, as a %, reported apart from the general average
- Build it
- Build a query set of account numbers, order IDs, and dates. Cosine similarity rewards topical resemblance. It does not reward an exact identifier match.
- Catches
- A retrieval eval that passes cleanly on topical queries and fails the moment a real user asks about a specific account.
- Evidence
- The identifier-query set and its pass rate, reported as its own line, never folded into the topical average.
- Measures
- Groundedness rate, as a %
- Build it
- An LLM-judge or NLI-style check comparing each claim in the answer to the specific source span it should trace back to.
- Catches
- A hallucinated answer dressed up around real-looking retrieved context.
- Evidence
- The groundedness scorer output, alongside the source span it checked each claim against.
- Measures
- Index-staleness lag, as the time between a source change and the index reflecting it
- Build it
- Change-data-capture ingest instead of a timer-based recrawl, with a tombstone record for anything deleted at the source.
- Catches
- The index serving a document that was deleted from the source months ago.
- Evidence
- The freshness-lag metric, plus a sample tombstone applied against a real deletion.
04 Trajectory evals
Once a system takes more than one step, the run itself needs a grade. That grade is separate from the answer at the end. Did it terminate? How many steps did it take, which tools did it call, at what cost? A trajectory eval catches the run that technically reached a correct answer by way of three unnecessary tool calls and a near-infinite retry. That defect is invisible to the golden-set layer above, because it only grades the final output.
- Measures
- Terminal-state rate, as a % split across success / failure / escalated
- Build it
- A forced terminal state, enforced by the harness on a step, token, and wall-clock budget. The model never decides this for itself.
- Catches
- An agent that runs for 40 minutes and produces nothing.
- Evidence
- The terminal-state distribution across a sample of recent runs.
Score whether the agent called the right tool for a scripted scenario, beyond whether it finished.
Pre-release- Measures
- Tool-choice accuracy, as a %
- Build it
- A labelled set of scripted multi-step scenarios with the expected tool sequence, graded against the sequence the agent called.
- Catches
- An agent that reaches a plausible answer by way of the wrong tool, or skips one the task needed.
- Evidence
- The scenario set with expected vs. actual tool sequences side by side.
- Measures
- Steps per completed task, as a distribution: median and p95
- Build it
- Log step count per run and compare it against the ceiling set in the run's bounded-loop budget.
- Catches
- A retry loop hiding inside a run that technically completed.
- Evidence
- The step-count distribution, reported alongside the configured ceiling.
- Measures
- $ per completed task
- Build it
- Divide total spend for the scenario set by the count of runs that reached a terminal success. Do not divide by the number of API calls made.
- Catches
- A retry loop hiding inside a good-looking per-call average.
- Evidence
- The cost-per-task figure, reported alongside the calls-to-completions ratio.
05 Online signals
No offline set, however carefully built, covers every input that production sends to the system. This is the layer that measures what happens once it does. It tracks task completion, escalation to a human, and how much a person had to edit an accepted answer. It also tracks cost, latency, and whether any of those numbers drift over time. This is production truth. It is not a proxy.
- Measures
- Task completion rate, as a %
- Build it
- Instrument the production flow for a defined 'completed' event, tracked separately from the offline eval's pass/fail condition.
- Catches
- A system that passes every offline eval and still fails in ways the eval set never anticipated.
- Evidence
- The production completion-rate dashboard, trended over time.
- Measures
- Escalation rate, as a %, broken out by step
- Build it
- Log every handoff event with the specific step it occurred at, beyond a running total.
- Catches
- A quality regression that shows up as users leaving for a human. It shows up before a lower eval score.
- Evidence
- The escalation-rate trend, segmented by the step in the flow it fires from.
- Measures
- Edit distance, as a normalized % change
- Build it
- Capture the final accepted version wherever a human can edit the model's draft before it ships, and diff it against the original.
- Catches
- A model whose answers 'pass' the eval suite but still need a heavy rewrite before anyone will use them.
- Evidence
- The edit-distance trend over time.
- Measures
- Drift, as the delta between the last CI pass rate and a live-traffic re-score
- Build it
- Periodically re-run the golden-set judge against sampled live traffic and diff the result against the CI number.
- Catches
- A provider-side model upgrade that silently changes quality between merges.
- Evidence
- The drift report comparing the CI score against the live re-score.
06 Adversarial & safety
A system can pass every quality check above and still do something it should not do, because none of those checks tried to attack it. This layer runs the labelled, adversarial equivalent of the golden set: injected instructions, known jailbreak patterns, PII-extraction attempts. It runs against the same tool-enabled path a real user reaches. It answers a narrower question than the rest of the stack.
- Measures
- Injection-resistance rate, as a %
- Build it
- Plant adversarial instructions inside retrieved content and tool outputs, beyond the user-facing prompt.
- Catches
- A document in your own knowledge base that reads like a command, and gets treated like one.
- Evidence
- The injection test set and its pass rate, tracked in CI alongside the quality suite.
- Measures
- Jailbreak-resistance rate, as a %
- Build it
- Maintain a running set of public jailbreak techniques. Add any that worked against this system before. Re-test on every guardrail or system-prompt change.
- Catches
- A guardrail that stopped working after a prompt edit nobody re-tested it against.
- Evidence
- The jailbreak test set and its pass rate, versioned against the system-prompt revision it ran on.
- Measures
- PII-leakage rate, as a %
- Build it
- A labelled set of prompts designed to extract PII from context the system has access to, checked against what comes back.
- Catches
- A retrieval or memory feature handing back personal data outside its intended scope.
- Evidence
- The leakage test set and its pass rate, plus the redaction rule each case is checked against.
- Measures
- Tool-abuse block rate, as a %
- Build it
- Adversarial scenarios targeting every tool that writes, sends, or deletes, confirming the human checkpoint fires or the call is rejected outright.
- Catches
- An agent talked into issuing a refund or sending a message it shouldn't have.
- Evidence
- The scenario set, plus a trace showing the checkpoint firing on a live example.
No checks match that combination. Clear a filter to see more.
Show, don't just state
A check bank entry, as one eval case.#
One case, wrong then right: a substring guess against deterministic checks with the judge scoped to tone.
The same eval case, two ways.
One case checks a substring and asks a judge whether the answer is good. The other checks the contract first, and saves the judge for tone.
What breaks refund_policy_question.yaml
case: refund_policy_question
input: "How do refunds work?"
assert:
- type: contains
value: "30 days"
judge_prompt: "Is this a good answer?" A single substring assertion, and a judge prompt that asks an open, unscoped question with no rubric behind it.
What works refund_policy_question.yaml
case: refund_policy_question
input: "How do refunds work?"
assert:
- type: schema
value: refund_answer.schema.json
- type: cites_source
value: refund-policy-v3
- type: tool_called
value: get_refund_policy
judge:
rubric: tone_matches_support_voice
scope: tone_only
weight: advisory Deterministic assertions on the schema, the citation, and the tool call, with the judge scoped to tone only.
- The response shape is checked against a schema before anything else runs. Schema validation rate
- The right tool call is checked as a deterministic assertion, not left to a judge's impression. Tool-call argument check
- A judge is a tool with a scope, not a verdict on the whole answer. Five ways a judge lies
Five ways a judge lies
LLM-as-judge is a tool.#
Every one of these produces a wrong number that looks right. None looks broken.
Self-preference
Looks like: A judge from the same model family as the generator rates that family's output higher across the board. It gives no stated reason.
Guardrail: Use a judge from a different model family than the generator, or budget for periodic cross-family spot checks.
Position bias
Looks like: In a side-by-side comparison, the judge favors whichever answer it saw first, or second, regardless of content.
Guardrail: Randomize answer order per comparison and average both orderings before trusting a verdict.
Verbosity bias
Looks like: The longer answer wins even when it says the same thing at greater length.
Guardrail: Score against a rubric that names length as a non-factor explicitly, and spot-check whether shorter correct answers are losing to longer ones.
Rubric drift
Looks like: The same rubric, applied a month apart, grades a static test case differently. The judge model or its default settings changed underneath you.
Guardrail: Pin the judge model and version; treat a judge upgrade as a breaking change that needs re-baselining. It is not a free improvement.
No human-calibration baseline
Looks like: The judge's pass rate looks stable, and nobody has checked it against a human rater in months.
Guardrail: Re-run a human-labelled holdout against the judge periodically and confirm agreement, beyond internal consistency.
The scoreboard
Six numbers, and how each one lies.#
Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.
Golden-set pass rate
- Why it matters
- Says whether the cases you already know matter still work after a change.
- How it misleads
- A rising rate against a small or unchanged set can mean the product improved. It can also mean the set no longer catches anything.
- Pair with
- Set coverage
Set coverage
- Why it matters
- How much of the real input distribution the golden set represents, including the edge cases that already caused an incident.
- How it misleads
- A large case count says nothing if the set is all easy, topical cases. A small set of hard-won incident cases can be worth more.
- Pair with
- Golden-set pass rate
Recall@k
- Why it matters
- Whether the retriever hands the generator the right context at all, measured before generation quality can hide the gap.
- How it misleads
- A recall number can look fine on topical queries. It can still miss on identifier or date queries when you do not score those separately.
- Pair with
- Groundedness rate
Groundedness rate
- Why it matters
- Whether the generated answer is traceable to the retrieved context, beyond being fluent and topically plausible.
- How it misleads
- A high groundedness score paired with low recall means the model repeats context that was wrong to begin with.
- Pair with
- Recall@k
p95 latency
- Why it matters
- The tail a user notices. A good median can hide a slow tail that drives complaints and abandonment.
- How it misleads
- Watching only the median lets a growing p95 hide in an average that still looks fine.
- Pair with
- Cost per completed task
Cost per completed task
- Why it matters
- What a feature costs to run per real outcome. Do not measure it per API call.
- How it misleads
- Cost per call undercounts it. A retry loop, or a fallback to a pricier model on failure, vanishes entirely from a per-call number.
- Pair with
- p95 latency
What this is built from
Verifiable proof.#
This page has no invented benchmark numbers and no client names. It uses what this site already documents.
- Golden-set gate in CI and trace-first observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. are patterns this site already documents in full. This page connects them into one stack instead of treating either as a standalone technique. Pattern: Golden-set gate in CI
- Judge-prefers-its-own-output, similarity-is-not-relevance, silent-tool-failure, and vibes-based-prompt-regression are named failure modes on this site. They are real. Each is the reason a specific layer above exists. Failure mode: Judge prefers its own output
- CodeHerder runs a golden-set gate in CI on every task branch before it merges. That is layer 2 of this stack, applied across an entire agent fleet instead of one feature. CodeHerder
- CostMon is in build. It will answer the layer-4 and scoreboard question, cost per completed task. It gives a self-serve number and removes the monthly reconciliation project. CostMon (coming soon)
- The eval methodology on Capabilities is the short version of this argument; this page is the full stack behind it. Capabilities: Evals methodology
Sources
- Stanford CRFM - Holistic Evaluation of Language Models (HELM)
The benchmark suite our check bank draws its evaluation method from. Retrieved 2026-08-24.
- NIST - AI Risk Management Framework (AI RMF 1.0)
The risk-management functions our governance and adoption controls map onto. Retrieved 2026-08-24.
- NIST - Generative AI Profile (NIST AI 600-1)
The generative-AI extension to the framework our eval and security controls reference. Retrieved 2026-08-24.
Not sure which layer you're missing
A Ship Audit runs this stack against your specific system and tells you which layers are live risks. It does not list the layers that are only incomplete in theory.
Questions
Before you build the suite.#
What teams ask us before they scope an eval stack.
01 How big does a golden set need to be?
50–100 real cases is a reasonable starting bar, pulled from production traffic and past incidents rather than invented. See the golden-set layer above. A held-out slice the prompt author never sees matters more than raw count.
Link to this answer: How big does a golden set need to be?02 Do we need all six layers on day one?
No. Start with contract checks and a golden-set gate in CI. Both are cheap and catch the two failure modes that do the most damage. Add trajectory and adversarial evalsAn eval suite is a versioned set of test cases that measures whether outputs are good. once there's an agent and real tool access worth protecting.
Link to this answer: Do we need all six layers on day one?03 Can we use an LLM to judge our own output?
Yes, with guardrails: pin the judge model and version, randomize comparison order, and calibrate against a human-labelled holdout periodically. See the five ways a judge lies above; each one is a way a judge score can look fine while being quietly wrong.
Link to this answer: Can we use an LLM to judge our own output?04 What's the difference between an eval and observability?
Evals catch a regression before it ships. Observability, the online-signals layer, catches the one that got through anyway. Neither replaces the other.
Link to this answer: What's the difference between an eval and observability?05 Isn't this a lot of infrastructure for a small feature?
Scope it to what the feature can do. A read-only summarizer needs few of the retrieval and trajectory layers. A system that sends messages or issues refunds needs the full stack, with adversarial checks.
Link to this answer: Isn't this a lot of infrastructure for a small feature?06 How is this different from the eval methodology on Capabilities?
That page is the three-step summary. This is the full stack behind it. It has six layers, 24 named checks, the ways a judge lies, and the numbers worth putting on a dashboard.
Link to this answer: How is this different from the eval methodology on Capabilities?