Six surfaces, 24 named controls, five ways a release process lies, and the scoreboard that keeps them honest. A code release has an obvious boundary; an AI system's behavior can change without a line of code moving, and this is the discipline for shipping that change on purpose.
You can't diff the output of a model change the way you diff a function's return value, which is why a flat percentage rollout and a git revert both fall short here. The model, the prompt, the retrieval index, the tool schemas, and the thresholds gating all of it are five independent moving parts, and any one of them can change behavior on its own. This page is the discipline for treating that seriously: what counts as a release, how it earns exposure, and what "revert" actually has to mean once the index was rebuilt in place.
The Handbook's Operate stage covers detecting a failure once a system is live. This page is what happens before that: how a change actually ships, so fewer failures reach production ramped to 100% before anyone notices.
Each surface fails a different way. None of them substitute for each other.
Covers: Every input that changes system behavior: model version, system prompt, retrieval corpus and index build, tool set and schemas, thresholds and config, and the harness or orchestration code around all of it.
Breaks when: An index gets rebuilt in place on a cron job nobody calls a deploy, so a behavior change ships with none of the review, the gate, or the rollback plan an actual release would get.
Watch: Share of behavior-changing inputs treated as releases
Covers: The checks a change has to clear before any real user sees it: a golden-set eval gate in CI, an offline replay against recorded production traffic, a cost and latency budget check, and a sign-off record naming what was actually compared against what.
Breaks when: The gate is green, and the golden set it ran against hasn't changed in months, so a green light means the change didn't break the cases everyone already agreed it wouldn't.
Watch: Golden-set freshness (days since a new case was added)
Covers: How a change reaches real users gradually instead of all at once: a canary cohort picked for blast radius, a ramp schedule with abort criteria written down first, a permanent holdback for comparison, and a kill switch that actually gets pulled on a schedule.
Breaks when: A canary is watched only for hard errors, so a quality regression that never throws an exception rides the ramp all the way to 100% of traffic before anyone notices the answers got worse.
Watch: Canary-to-full ramp duration
Covers: What "revert" actually has to mean once the model, the prompt, the index and the tool schemas can each change independently: pinned model versions, one immutable bundle per release, a stated rule for forward-fix versus rollback, and a provider deprecation calendar with the migration dated before the shutoff.
Breaks when: A rollback restores last week's code but not the index it was built against, so the "reverted" system answers from an index that no longer matches the prompt reading it.
Watch: Reversal MTTR
Covers: What gets monitored once a change is live: guard metrics (refusal rate, tool-error rate, p95 latency, spend per request) tracked separately from quality metrics, sampled human review of live traffic, drift thresholds with a named owner, and the trace plumbing all of it actually reads from.
Breaks when: A dashboard tracks refusal rate and latency, both look fine, and the system has been quietly answering worse questions worse for two weeks because nothing on that dashboard measures correctness.
Watch: Drift-detection lag
Covers: What happens once something goes wrong: a severity ladder built for a system that fails by being confidently wrong rather than by crashing, a runbook whose first step is to pin and reduce exposure, a stated disclosure path, and a rule that every postmortem becomes a new eval case.
Breaks when: A team reports "zero incidents" for the quarter, and the honest reading is that nothing was measured well enough for anyone to recognize one when it happened.
Watch: Declared incidents with a resulting eval case
Filter by how often it's exercised (continuous, periodic, triggered), then copy the visible list as a Markdown checklist.
Showing all 24 controls
A code release has an obvious boundary: a commit merges, a build runs, a version ships. An AI system's behavior can change without a single line of code moving: a provider quietly updates a model behind an alias, a nightly job rebuilds the retrieval index, someone edits a threshold in a config panel. None of those look like a release, and none of them get a release's scrutiny, which is exactly why this surface exists: name every input that can change what the system does, on purpose, before deciding how any of it ships.
A code review asks whether a diff is correct. An AI release gate has to ask a harder question: does this change make the system worse on inputs nobody wrote a test for. The eval stack answers that for known cases; this surface is what turns a passing eval run into a real go/no-go decision, backed by a record of what was compared, replayed against traffic the system actually saw, and checked against a budget before the invoice makes the case for you.
You cannot diff the output of a model change the way you diff a function's return value, which is exactly why a flat percentage rollout is the wrong shape for it: 5% of traffic tells you almost nothing if you were only ever going to look at the error rate. This surface is what makes exposure genuinely gradual: a cohort chosen because it bounds the damage, a written abort rule instead of a judgment call made mid-incident, a holdback that lets you compare against what would have happened anyway, and a kill switch that has actually been pulled before the day it matters.
Reverting a normal service means redeploying the last good binary. Reverting an AI system means asking which of five independent things actually needs to go back: the model, which a provider might have moved out from under you; the prompt; the index the retrieval layer reads; the tool schemas an agent calls against; the thresholds gating all of it. Treat any one of those as reversible on its own and the other four keep drifting underneath it. This surface is the discipline that makes "roll back" resolve to one clean, tested action instead of a scramble to figure out what state the system was actually in.
A guard metric and a quality metric answer different questions, and conflating them is how a rollout looks healthy while it's actively getting worse: p95 latency and tool-error rate tell you the system is still running, not that it's still right. This surface keeps those two families separate, adds a human in the loop sampling real traffic instead of trusting the dashboard alone, and puts a named owner on a drift threshold instead of a metric nobody is actually accountable for. None of it works without a trace to read from, which is the piece every other control here depends on.
A traditional outage announces itself: the service is down, the alert fires, everyone knows it's an incident. An AI system's worst failures often look like a normal response, just a wrong one, delivered with the same confidence as a right one. A severity ladder built around crashes and downtime will never catch that, which is why this surface starts by defining severity around being wrong instead of being unavailable, then makes the first runbook step reduce exposure rather than diagnose, and closes the loop by turning every real incident into a case the eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. will catch next time.
No controls match that combination. Clear a filter to see more.
Every one of these looks like a working release process right up until the release it was supposed to catch.
Looks like: CI shows a passing eval run on every release, month after month, and nobody has touched the golden set in a long while.
Costs you: A green gate stops meaning "this didn't break anything we care about" and starts meaning "this didn't break the specific cases we tested four months ago," which is a much weaker claim wearing the same badge.
Fix: Add a real, checked freshness expectation on the golden set itself, and treat every closed incident as an obligation to add a case, a higher bar than a document to file away.
See the control: A golden-set eval gate blocks a regressing release →Looks like: A rollout ramps cleanly through its scheduled steps because the only thing anyone is watching is the error rate, and the error rate never moves.
Costs you: A quality regression that never throws an exception rides the ramp all the way to 100% of traffic, fully invisible to the one metric anyone was actually checking.
Fix: Track guard metrics and quality metrics as two separate families from the start, and gate ramp progression on both, not on whichever one is easiest to instrument.
See the control: Track guard metrics separately from quality metrics →Looks like: An incident gets a fast, confident rollback: the previous prompt and model version are back within minutes.
Costs you: The index kept rebuilding on its own schedule the whole time, so the "reverted" system is now running a prompt built against one index, reading from a different one, and the answers are wrong in a new way nobody planned for.
Fix: Bundle prompt, index and tool schemas as one versioned unit, and make reverting the bundle the only rollback path, so there is no way to revert one piece without the others.
See the control: Ship prompt, index and tool schemas as one immutable bundle →Looks like: The runbook names a clear, one-step way to cut exposure instantly, and it reads as reassuring right up until someone actually needs it.
Costs you: A kill switch that has never been exercised is a hypothesis about what would happen, not a tested capability, and the incident it's needed for is the worst possible moment to discover the hypothesis was wrong.
Fix: Put the kill switch on the same scheduled-drill discipline as a backup restore test, and log the result every time. The first drill proves nothing about the tenth.
See the control: Exercise the kill switch on a schedule →Looks like: A quarterly report proudly states zero AI incidents, and the team genuinely believes it, because nothing tripped the alerting they have in place.
Costs you: A system that fails by being confidently wrong instead of by crashing can run that way for a long time under a severity ladder built for outages, producing a real absence of harm and a fake absence of failures that look identical from the dashboard.
Fix: Build the severity ladder around wrong-but-plausible output specifically, and pair it with sampled human review that would actually surface the failure a metric alone would miss.
See the control: Build a severity ladder for a system that fails by being confidently wrong →Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.
No invented reliability stats. Just what's already documented on this site, and how it connects.
The research programme behind the delivery metrics our release and delivery pages measure. Retrieved 2026-08-24.
The method for setting a service-level objective our release scoreboard follows. Retrieved 2026-08-24.
The error-budget policy pattern our release gate borrows. Retrieved 2026-08-24.
A Ship Audit checks this release path against your specific system and tells you which control is real, versus which one has never been exercised.
What teams ask us before they build a real release process for an AI system.
Mechanically, much of it looks like ordinary CI/CD: a gate, a ramp, a rollback. The part that's genuinely different is that you can't diff the output of a model change the way you diff a function's return value, so the gate has to run an eval suite instead of a test suite, and reversal has to account for an index and a prompt that drift independently of the code.
Link to this answer: Isn't this just CI/CD with different names?If the prompt change can reach real users without going through a gate or a rollback path, then yes, treat it as a release. The blast radius of a bad prompt edit is exactly the same as a bad code deploy; the only thing missing is usually the process, not the risk.
Link to this answer: We just edit the prompt in a dashboard. Do we really need a release process for that?Operate is about detecting a failure once a system is already live: unbounded loops, silent tool failures, a stale index. This page is about the mechanics of shipping the change in the first place, how it gets gated, ramped and reversed, so fewer of those failures reach production ramped to 100% before anyone notices.
Link to this answer: What's the difference between this and the Operate stage in the handbook?You don't roll back a rebuild in place; you version the index build itself, the same way you'd version a model, and pin the previous build as the fallback. That's the whole argument for shipping prompt, index and tool schemas as one bundle: reverting means switching back to a previous, complete bundle, not trying to undo a rebuild after the fact.
Link to this answer: How do you roll back a retrieval index that already got rebuilt?A harmful or materially wrong answer reaching a real user, at scale or in a high-stakes context, counts the same as an outage would. The severity ladder here starts from confidently wrong output rather than from downtime, because that's the failure mode an AI system is actually prone to.
Link to this answer: What counts as a severity-1 release incident if the system never actually crashes?No. A golden-set gate and a documented rollback path cost little and catch most of the risk on a first release. Scale the rest, the canary cohort discipline, the drift thresholds, the incident ladder, to how much exposure and how many independent moving parts (model, prompt, index, tools) the system actually has.
Link to this answer: Do we need all six surfaces before we ship anything?Source: https://customlabs.io/release/