Evals

The Eval Stack: how to know an AI system actually works.

Six layers, 24 named checks, five ways an LLM judge lies, and the numbers worth putting on a dashboard. This is the discipline behind the evaluate stage of the Handbook, not a pitch for a tool.

A demo proves a system can work once, on the cases you picked to show it off. An eval proves it works on the cases you don't get to pick: the ones a real user actually sends, including the ones that already broke it before. Most teams ship on vibes, discover the regression in production, and then can't say whether the fix helped, because nothing was measuring in the first place.

Evaluation isn't one gate. It's a stack of layers, and each one catches a class of failure the layer below it is blind to. Skipping a layer doesn't lower your score. It means you stop measuring that class of failure entirely, and find out about it in production instead.

The stack

Six layers, stacked, not gated.

Each layer catches something the one below it can't see. None of them replace each other.

Contract checks

Catches: Malformed output — broken JSON, a missing field, a tool call with the wrong argument shape.

Blind to: Whether a well-formed answer is actually correct.

Runs when: Every request, inline, before the response is used

Golden-set task evals

Catches: A prompt, model, or retrieval change that quietly makes real cases worse.

Blind to: Retrieval quality in isolation, and any case shape the set doesn't include.

Runs when: Every change to a prompt, model, or retrieval config, in CI

Retrieval evals

Catches: The retriever handing the generator the wrong context, or none at all.

Blind to: How well the generator uses good context once it actually has it.

Runs when: Alongside golden-set evals, scored as its own number

Trajectory evals

Catches: An agent that loops, calls the wrong tool, or never terminates.

Blind to: Whether the final answer, once it stops, was any good.

Runs when: Pre-release, against scripted multi-step scenarios

Online signals

Catches: Everything the offline suite never saw, because this is real traffic.

Blind to: Anything that fails silently and never gets reported or escalated.

Runs when: Continuous, on live production traffic

Adversarial & safety

Catches: Prompt injection, jailbreaks, PII leakage, and tool misuse — under deliberate attack.

Blind to: Ordinary quality; it only answers whether the system can be made to misbehave on purpose.

Runs when: Pre-release, and after any change to tools or the system prompt

The check bank

24 checks, six layers.

Filter by when it runs (pre-merge, pre-release, continuous), then copy the visible list as a Markdown checklist.

01 Contract checks

The cheapest layer and the one most teams already half-have: does the response even parse. A schema check can't tell you the answer is right, only that it's shaped the way the next piece of code expects — but a wrongly-shaped response breaks that next step regardless of whether the content was good, so it runs on every single request, not just the ones in a test suite.

Validate every structured response against its schema before it reaches downstream code.

Continuous

Measures: Schema-pass rate, as a %
Build it: A JSON Schema or typed parser on every structured output; reject and surface a distinguishable error, not a silent retry loop.
Catches: A malformed response breaking the next step in the chain.
Evidence: The schema definitions plus the validation-failure rate, tracked separately from execution failures.

Glossary: Structured output

Validate every tool-call argument against a typed contract, not a free-form string.

Continuous

Measures: Invalid tool-call rate, as a %
Build it: Enums and patterns enforced server-side; a mutating call has to trace back to a value an earlier read actually returned, never one the model invented.
Catches: Tool argument hallucination — a plausible-looking ID the model made up because it couldn't recall the real one.
Evidence: The tool schema definitions plus the validation-failure log.

Pattern: Typed tool contract Failure mode: Tool argument hallucination

Give every tool response an explicit status field the model can branch on.

Continuous

Measures: Silent-failure rate — tool calls that returned success with nothing usable inside — as a %
Build it: A typed status enum (ok / empty / error) instead of HTTP 200 with a failure buried in the response body.
Catches: Silent tool failure — the model reporting an absence of data as a fact because nothing told it otherwise.
Evidence: The tool response schema, plus a sample of an error actually surfaced as status: error, not a 200.

Failure mode: Silent tool failure

Track what share of production traffic actually exercises the schema check, not just what CI covers.

Continuous

Measures: Contract-check coverage, as a % of live requests
Build it: Log a validation pass/fail on every production request, not only the request shapes a test author thought to write.
Catches: A schema gap on a request shape nobody wrote a test case for.
Evidence: The coverage dashboard, broken out by request type.

02 Golden-set task evals

The layer most people mean when they say "we have evals": a fixed, labelled set of real cases, each with a checkable pass/fail condition, run as a required CI gate. It answers "did this change help or hurt" with a number instead of a re-run of whichever two examples happened to be open in a tab. It says nothing about a case shape the set never included — which is exactly why the next four layers exist.

Seed the golden set from production traces, not from the prompts you wrote while building.

Pre-merge

Measures: Set size, as a count of labelled cases (50–100 to start)
Build it: Pull real traces plus every past incident case; label each with a specific pass/fail condition, not a description of the desired vibe.
Catches: An eval suite that only ever re-tests the happy path its own author already knew would pass.
Evidence: The labelled case set, with provenance — production trace vs. authored — tagged per case.

Insight: Evals before you ship

Hold out a subset of the set the prompt author never sees.

Pre-merge

Measures: Held-out pass rate vs. visible-set pass rate, as two separate percentages
Build it: Split the set at creation time; only a reviewer, never the person iterating on the prompt, can see the held-out half.
Catches: A prompt tuned against the same cases used to grade it — the eval-suite version of training on the test set.
Evidence: The set-split record and both pass rates reported side by side, never blended into one number.

Run the golden set on every prompt, model, or retrieval change, and fail the merge on a regression.

Pre-merge

Measures: Pass rate, as a %, tracked per commit
Build it: A required CI check, not an optional script — the same gate a unit-test suite would get, applied to a prompt change.
Catches: A regression that ships because nobody happened to re-run the eval by hand that day.
Evidence: The CI configuration plus the pass-rate history across recent merges.

Pattern: Golden-set gate in CI Failure mode: Vibes-based prompt regression

Tag and track what share of the set is edge cases that already caused an incident, not routine happy-path examples.

Pre-merge

Measures: Edge-case share of the set, as a %
Build it: Tag each case by origin — routine, edge case, past incident — at label time, and report the mix alongside the raw count.
Catches: A set that looks comprehensive by count but is quietly all easy cases.
Evidence: The case-origin breakdown, reported next to the pass rate it produced.

03 Retrieval evals

Retrieval and generation fail independently, so they need separate scores or a regression in one hides behind a passing average from the other. Recall@k, precision, and groundedness are measured against the retrieved chunk itself, before the generator ever gets a turn — the only way to tell "the model wrote a bad answer" apart from "the model wrote a good answer to the wrong context."

Score recall@k separately from generation quality.

Pre-merge

Measures: Recall@k, as a %
Build it: Label the correct source chunk per query, then check whether it's in the top-k retrieved set before the generator ever sees it.
Catches: A generator that sounds confident on context that never contained the answer in the first place.
Evidence: The recall@k report, scored independently of the generation eval it feeds.

Failure mode: Similarity is not relevance

Evaluate identifier and date-shaped queries as their own set, separate from topical queries.

Pre-merge

Measures: Identifier-query pass rate, as a %, reported apart from the general average
Build it: Build a query set of account numbers, order IDs, and dates specifically — cosine similarity rewards topical resemblance, not an exact identifier match.
Catches: A retrieval eval that passes cleanly on topical queries and fails the moment a real user asks about a specific account.
Evidence: The identifier-query set and its pass rate, reported as its own line, never folded into the topical average.

Score whether the generated answer is actually attributable to the retrieved context, not just topically related to it.

Pre-merge

Measures: Groundedness rate, as a %
Build it: An LLM-judge or NLI-style check comparing each claim in the answer to the specific source span it should trace back to.
Catches: A hallucinated answer dressed up around real-looking retrieved context.
Evidence: The groundedness scorer output, alongside the source span it checked each claim against.

Glossary: Hallucination

Track index freshness against the source, with a defined tombstone path for deletions.

Continuous

Measures: Index-staleness lag, as the time between a source change and the index reflecting it
Build it: Change-data-capture ingest instead of a timer-based recrawl, with a tombstone record for anything deleted at the source.
Catches: The index serving a document that was deleted from the source months ago.
Evidence: The freshness-lag metric, plus a sample tombstone applied against a real deletion.

Failure mode: Stale index serves deleted content

04 Trajectory evals

Once a system takes more than one step, the run itself becomes something to grade, separately from the answer at the end of it: did it terminate, how many steps did it take, which tools did it call, at what cost. A trajectory eval catches the run that technically reached a correct answer by way of three unnecessary tool calls and a near-infinite retry — a defect the golden-set layer above never sees, because it only grades the final output.

Track what share of agent runs terminate in a defined state instead of looping.

Pre-release

Measures: Terminal-state rate, as a % split across success / failure / escalated
Build it: A forced terminal state enforced by the harness on a step, token, and wall-clock budget — never left to the model to decide for itself.
Catches: An agent that runs for 40 minutes and produces nothing.
Evidence: The terminal-state distribution across a sample of recent runs.

Pattern: Bounded agent loop Failure mode: Unbounded agent loop

Score whether the agent called the right tool for a scripted scenario, not just whether it finished.

Pre-release

Measures: Tool-choice accuracy, as a %
Build it: A labelled set of scripted multi-step scenarios with the expected tool sequence, graded against the sequence the agent actually called.
Catches: An agent that reaches a plausible answer by way of the wrong tool, or skips one the task actually needed.
Evidence: The scenario set with expected vs. actual tool sequences side by side.

Track step count per completed task against a stated ceiling.

Pre-release

Measures: Steps per completed task, as a distribution — median and p95
Build it: Log step count per run and compare it against the ceiling set in the run's bounded-loop budget.
Catches: A retry loop hiding inside a run that technically completed.
Evidence: The step-count distribution, reported alongside the configured ceiling.

Report cost per completed task, not per call.

Pre-release

Measures: $ per completed task
Build it: Divide total spend for the scenario set by the count of runs that reached a genuine terminal success — not by the number of API calls made.
Catches: A retry loop hiding inside a good-looking per-call average.
Evidence: The cost-per-task figure, reported alongside the calls-to-completions ratio.

Failure mode: Retry-amplified spend

05 Online signals

No offline set, however carefully built, covers every input production will actually throw at the system — this is the layer that measures what happens once it does. Task completion, escalation to a human, how much a person had to edit an accepted answer, cost and latency per task, and whether any of those numbers are drifting over time: production truth, not a proxy for it.

Track task completion rate on live traffic, not just the offline pass rate.

Continuous

Measures: Task completion rate, as a %
Build it: Instrument the production flow for a defined 'completed' event, tracked separately from the offline eval's pass/fail condition.
Catches: A system that passes every offline eval and still fails in ways the eval set never anticipated.
Evidence: The production completion-rate dashboard, trended over time.

Track escalation and handoff rate to a human, and where in the flow it happens.

Continuous

Measures: Escalation rate, as a %, broken out by step
Build it: Log every handoff event with the specific step it occurred at, not just a running total.
Catches: A quality regression that shows up as users bailing out to a human, before it ever shows up as a lower eval score.
Evidence: The escalation-rate trend, segmented by the step in the flow it fires from.

Measure edit distance between the model's draft and what a human actually accepted or sent.

Continuous

Measures: Edit distance, as a normalized % change
Build it: Capture the final accepted version wherever a human can edit the model's draft before it ships, and diff it against the original.
Catches: A model whose answers 'pass' the eval suite but still need a heavy rewrite before anyone will use them.
Evidence: The edit-distance trend over time.

Re-run the offline eval metrics against a sample of live traffic on a schedule, not just at merge time.

Continuous

Measures: Drift, as the delta between the last CI pass rate and a live-traffic re-score
Build it: Periodically re-run the golden-set judge against sampled live traffic and diff the result against the CI number.
Catches: A provider-side model upgrade that silently changes quality between merges.
Evidence: The drift report comparing the CI score against the live re-score.

Glossary: Observability

06 Adversarial & safety

A system can pass every quality check above and still be talked into something it shouldn't do, because none of those checks were trying to attack it. This layer runs the labelled, adversarial equivalent of the golden set — injected instructions, known jailbreak patterns, PII-extraction attempts — against the same tool-enabled path a real user reaches, and answers a narrower question than the rest of the stack on purpose.

Run a labelled set of prompt-injection attempts through the full tool-enabled path.

Pre-release

Measures: Injection-resistance rate, as a %
Build it: Plant adversarial instructions inside retrieved content and tool outputs, not only in the user-facing prompt.
Catches: A document in your own knowledge base that reads like a command, and gets treated like one.
Evidence: The injection test set and its pass rate, tracked in CI alongside the quality suite.

Failure mode: Injection via retrieved content Glossary: Red-teaming

Run known jailbreak patterns against the system prompt and guardrails.

Pre-release

Measures: Jailbreak-resistance rate, as a %
Build it: Maintain a running set of public jailbreak techniques plus any that have worked against this system before, and re-test on every guardrail or system-prompt change.
Catches: A guardrail that quietly stopped working after a prompt edit nobody re-tested it against.
Evidence: The jailbreak test set and its pass rate, versioned against the system-prompt revision it ran on.

Test whether the system will surface personal data it shouldn't, on request.

Pre-release

Measures: PII-leakage rate, as a %
Build it: A labelled set of prompts designed to extract PII from context the system has access to, checked against what actually comes back.
Catches: A retrieval or memory feature handing back personal data outside its intended scope.
Evidence: The leakage test set and its pass rate, plus the redaction rule each case is checked against.

Script scenarios that try to make the agent misuse a write-scoped tool.

Pre-release

Measures: Tool-abuse block rate, as a %
Build it: Adversarial scenarios targeting every tool that writes, sends, or deletes, confirming the human checkpoint fires or the call is rejected outright.
Catches: An agent talked into issuing a refund or sending a message it shouldn't have.
Evidence: The scenario set, plus a trace showing the checkpoint firing on a live example.

Pattern: Human checkpoint before irreversible actions

Five ways a judge lies

LLM-as-judge is a tool, not a verdict.

Every one of these produces a confidently wrong number, not an obviously broken one.

Self-preference

Looks like: A judge from the same model family as the generator rates that family's output a little higher, across the board, for no stated reason.

Guardrail: Use a judge from a different model family than the generator, or budget for periodic cross-family spot checks.

Position bias

Looks like: In a side-by-side comparison, the judge favors whichever answer it saw first — or second — regardless of content.

Guardrail: Randomize answer order per comparison and average both orderings before trusting a verdict.

Verbosity bias

Looks like: The longer answer wins even when it says the same thing at greater length.

Guardrail: Score against a rubric that names length as a non-factor explicitly, and spot-check whether shorter correct answers are losing to longer ones.

Rubric drift

Looks like: The same rubric, applied a month apart, grades a static test case differently — the judge model or its default settings changed underneath you.

Guardrail: Pin the judge model and version; treat a judge upgrade as a breaking change that needs re-baselining, not a free improvement.

No human-calibration baseline

Looks like: The judge's pass rate looks stable, and nobody has checked it against a human rater in months.

Guardrail: Re-run a human-labelled holdout against the judge periodically and confirm agreement, not just internal consistency.

The scoreboard

Six numbers, and how each one lies.

Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.

Golden-set pass rate

Why it matters: Says whether the cases you already know matter still work after a change.
How it misleads: A rising rate against a small or unchanged set can mean the product improved, or it can mean the set stopped being hard enough to catch anything.
Pair with: Set coverage

Set coverage

Why it matters: How much of the real input distribution — including the edge cases that already caused an incident — the golden set actually represents.
How it misleads: A large case count says nothing if the set is all easy, topical cases; a small set of hard-won incident cases can be worth more.
Pair with: Golden-set pass rate

Recall@k

Why it matters: Whether the retriever hands the generator the right context at all, measured before generation quality can hide the gap.
How it misleads: A recall number can look fine on topical queries and still miss badly on identifier or date-shaped ones, if those are not scored separately.
Pair with: Groundedness rate

Groundedness rate

Why it matters: Whether the generated answer is actually traceable to the retrieved context, not just fluent and topically plausible.
How it misleads: A high groundedness score paired with low recall just means the model is faithfully repeating context that was wrong to begin with.
Pair with: Recall@k

p95 latency

Why it matters: The tail a user actually notices — a good median hides a slow tail that drives complaints and abandonment.
How it misleads: Watching only the median lets a growing p95 hide in an average that still looks fine.
Pair with: Cost per completed task

Cost per completed task

Why it matters: What a feature actually costs to run per real outcome, not per API call.
How it misleads: Cost per call undercounts it — a retry loop or a fallback to a pricier model on failure vanishes entirely from a per-call number.
Pair with: p95 latency

What this is built from

Verifiable, not claimed.

No invented benchmark numbers, no client names. Just what's already documented on this site.

Golden-set gate in CI and trace-first observability are patterns this site already documents in full — this page connects them into one stack instead of treating either as a standalone technique. Pattern: Golden-set gate in CI
Judge-prefers-its-own-output, similarity-is-not-relevance, silent-tool-failure, and vibes-based-prompt-regression are named failure modes on this site, not hypotheticals — each is the reason a specific layer above exists. Failure mode: Judge prefers its own output
CodeHerder runs a golden-set gate in CI on every task branch before it merges — layer 2 of this stack, applied across an entire agent fleet instead of one feature. CodeHerder
CostMon is being built to answer the layer-4 and scoreboard question — cost per completed task — as a self-serve number instead of a monthly reconciliation project. CostMon (coming soon)
The eval methodology on Capabilities is the short version of this argument; this page is the full stack behind it. Capabilities: Evals methodology

Not sure which layer you're missing

A Ship Audit runs this stack against your specific system and tells you which layers are live risks, not just which ones are theoretically incomplete.

Book a Ship Audit → See the Evaluate stage →

Questions

Before you build the suite.

What teams ask us before they scope an eval stack.

01 How big does a golden set actually need to be?

50–100 real cases is a reasonable starting bar, pulled from production traffic and past incidents rather than invented — see the golden-set layer above. A held-out slice the prompt author never sees matters more than raw count.

02 Do we need all six layers on day one?

No — start with contract checks and a golden-set gate in CI. Both are cheap and catch the two failure modes that do the most damage. Add trajectory and adversarial evals once there's an agent and real tool access worth protecting.

03 Can we just use an LLM to judge our own output?

Yes, with guardrails — pin the judge model and version, randomize comparison order, and calibrate against a human-labelled holdout periodically. See the five ways a judge lies above; each one is a way a judge score can look fine while being quietly wrong.

04 What's the difference between an eval and observability?

Evals catch a regression before it ships; observability — the online-signals layer — catches the one that got through anyway. Neither replaces the other.

05 Isn't this a lot of infrastructure for a small feature?

Scope it to what the feature can do. A read-only summarizer barely needs the retrieval and trajectory layers; something that can send messages or issue refunds needs the full stack, adversarial checks included.

06 How is this different from the eval methodology on Capabilities?

That page is the three-step summary; this is the full stack behind it — six layers, 24 named checks, the ways a judge lies, and the numbers worth putting on a dashboard.