Release

The AI Release Path: ship a change, ramp it, and reverse it if it's wrong.

Six surfaces, 24 named controls, five ways a release process lies, and the scoreboard that keeps them honest. A code release has an obvious boundary. An AI system's behavior can change without a line of code moving. This is the discipline for shipping that change on purpose.

Updated November 2, 2026 First published August 13, 2026

34 min read

Markdown

You can't diff the output of a model change the way you diff a function's return value. A flat percentage rollout and a git revert both fall short here. The model, the prompt, the retrieval index, the tool schemas and the gating thresholds are five independent moving parts. Any one of them can change behavior on its own. This page covers what counts as a release and how a release earns exposure. It also defines what "revert" has to mean once the index was rebuilt in place.

The Handbook's Operate stage covers detecting a failure once a system is live. This page is what happens before that: how a change actually ships, so fewer failures reach production ramped to 100% before anyone notices.

The six surfaces

What a release process actually has to cover.#

Each surface fails a different way. None of them substitute for each other.

A release moves through six stages in order, from what counts as a release to the incident record. A reversal can trigger a new release cycle.

What counts as a release

Covers: Every input that changes system behavior. That covers model version, system prompt, retrieval corpus and index build, tool set and schemas, thresholds and config. It also covers the harness or orchestration code around all of them.

Breaks when: An index gets rebuilt in place on a cron job nobody calls a deploy. The behavior change ships without the review, the gate or the rollback plan a release would get.

Watch: Share of behavior-changing inputs treated as releases

The gate before exposure

Covers: The checks a change has to clear before any real user sees it. They are a golden-set eval gate in CI, an offline replay against recorded production traffic and a cost and latency budget check. A sign-off record names what was compared against what.

Breaks when: The gate is green, and the golden set it ran against has not changed in months. A green light then means only that the change broke none of the cases everyone already agreed on.

Watch: Golden-set freshness (days since a new case was added)

Progressive exposure

Covers: How a change reaches real users gradually instead of all at once. The pieces are a canary cohort picked for blast radius, a ramp schedule with abort criteria set first and a permanent holdback for comparison. A kill switch gets pulled on a schedule.

Breaks when: A canary is watched only for hard errors. A quality regression that throws no exception rides the ramp to 100% of traffic. Nobody notices the answers got worse.

Watch: Canary-to-full ramp duration

Reversal and pinning

Covers: What "revert" has to mean once the model, the prompt, the index and the tool schemas can each change independently. It means pinned model versions and one immutable bundle per release. It also means a stated rule for forward-fix versus rollback. A provider deprecation calendar dates the migration before the shutoff.

Breaks when: A rollback restores last week's code but not the index it was built against. The "reverted" system answers from an index that no longer matches the prompt reading it.

Watch: Reversal MTTR

In-flight watch

Covers: What gets monitored once a change is live. Guard metrics (refusal rate, tool-error rate, p95 latency, spend per request) are tracked separately from quality metrics. The watch also needs sampled human review of live traffic and drift thresholds with a named owner. The trace plumbing feeds all of it.

Breaks when: A dashboard tracks refusal rate and latency, and both look fine. The system has answered questions worse for two weeks. Nothing on that dashboard measures correctness.

Watch: Drift-detection lag

Incident and change record

Covers: What happens once something goes wrong. The severity ladder is built for a system that fails by being confidently wrong. The runbook's first step is to pin and reduce exposure. A stated disclosure path and a rule that every postmortem becomes a new eval case complete the set.

Breaks when: A team reports "zero incidents" for the quarter. The honest reading is that nothing was measured well enough to recognize one.

Watch: Declared incidents with a resulting eval case

The control bank

24 controls, six surfaces.#

Filter by how often it's exercised (continuous, periodic, triggered), then copy the visible list as a Markdown checklist.

01 What counts as a release

A code release has an obvious boundary: a commit merges, a build runs, a version ships. An AI system's behavior can change without a single line of code moving. A provider updates a model behind an alias. A nightly job rebuilds the retrieval index. Someone edits a threshold in a config panel. None of those look like a release. None of them get a release's scrutiny. This surface exists to name every input that can change what the system does. Name them on purpose, before you decide how any of them ships.

Name every input that counts as a release

Triggered

What it covers: An explicit, written list of everything that changes system behavior. It names the model version, system prompt, retrieval index build, tool set and schemas, thresholds and config, and the harness code around them.
How to build it: Write the list once per system. Treat any change to an item on it as a release, whichever team makes the change.
Produces: A shared answer to "did anything change" that doesn't depend on remembering every team that can touch behavior.
Tradeoff: A list this broad pulls config and index changes under process that used to move without any review at all. That friction is the point, not a bug in the list.
Prove it: Every behavior-changing input on the list has a release record attached to its last change. Check the record against the change history. Do not trust the list's own claim.

Treat an index rebuild as a release, not a maintenance job

Periodic

What it covers: Whether rebuilding the retrieval index, on a schedule or on demand, goes through the same release discipline as a model or prompt change.
How to build it: Route index rebuilds through the same gate and versioning as any other release input, instead of a cron job that runs unattended and unreviewed.
Produces: A rebuilt index that carries a version, a gate result, and a rollback path, the same as a code deploy.
Tradeoff: Gating every rebuild adds latency to what used to be a fire-and-forget job. Size the gate to the rebuild's actual blast radius rather than blocking a routine refresh on a full eval run.
Prove it: The last three index rebuilds each resolve to a release record, checked against the ingest job's own logs.

Treat a threshold or config change as a release

Continuous

What it covers: A rule that a changed confidence threshold, routing rule or guardrail setting goes through the same review a prompt change gets. No config panel edit goes unlogged.
How to build it: Route config and threshold edits through source control or an equivalent change log, with the same reviewer expectations as a code change.
Produces: A change history for the numbers that quietly decide how the system behaves, the same discipline the code around them already gets.
Tradeoff: This is real friction on values teams are used to tuning live. The alternative is a system whose actual behavior at any past date can't be reconstructed.
Prove it: Every threshold currently in production traces to a dated, reviewed change record.

Glossary: Model deprecation

Give every release one named owner

Continuous

What it covers: A single person accountable for a given release, distinct from whoever happened to trigger the underlying change.
How to build it: A required owner field on the release record, assigned before the change goes out, not reconstructed afterward from commit history.
Produces: A single name to ask when a release does something nobody expected.
Tradeoff: Ownership needs to survive a reorg or a departure. An owner field pointing at someone who left is worse than an honest blank.
Prove it: A release record picked at random names a current, reachable owner.

02 The gate before exposure

A code review asks whether a diff is correct. An AI release gate has to ask a harder question: does this change make the system worse on inputs nobody wrote a test for. The eval stack answers that for known cases. This surface turns a passing eval run into a go/no-go decision. The decision rests on a record of what was compared. It also rests on a replay against traffic the system saw, and on a budget check that comes before the invoice does.

A golden-set eval gate blocks a regressing release

Continuous

What it covers: A CI check that runs the labelled eval set against a candidate release and blocks it from merging or shipping if the pass rate drops.
How to build it: Wire the eval suite into CI as a required check. It gates a release the way a unit test suite gates a code change. Allow an explicit, reviewed override for a deliberate trade-off.
Produces: A release that can't reach production carrying a known regression nobody signed off on.
Tradeoff: A gate this strict blocks a release on a case that later turns out not to matter. That is a cheaper failure mode than shipping a regression the gate could have caught.
Prove it: The last ten merges to the release branch each show a passing, logged gate run, not an override with no reason attached.

Pattern: Golden-set gate in CI

Replay recorded production traffic before shipping

Periodic

What it covers: Running a candidate release against a sample of real, previously captured production requests, offline, before it ever serves a real user.
How to build it: Capture a rolling sample of production inputs, with the same privacy handling live traffic gets. Replay it against every candidate release as a required gate step.
Produces: Evidence about how a change behaves on the actual input distribution, a wider check than the golden set someone assembled once.
Tradeoff: A useful replay sample has to be refreshed as real traffic shifts, or it becomes its own stale golden set with an extra step.
Prove it: The replay sample was refreshed inside the last release cycle, checked against its own capture date.

Glossary: Shadow deployment

Check the cost and latency budget before exposure, not after the invoice

Continuous

What it covers: A gate step that measures a candidate release's cost per request and p95 latency against a stated budget, before it reaches any real traffic.
How to build it: Run the same replay sample used for the quality gate through a cost and latency measurement. Block a release that misses its budget by too wide a margin.
Produces: A release that can't ship a quiet 3x cost regression that would otherwise only show up on next month's bill.
Tradeoff: A budget set once and never revisited against real usage becomes exactly the stale check it was meant to replace.
Prove it: The budget for the current release line was checked against last month's actual spend, not the original launch estimate.

Record what was actually compared against what

Triggered

What it covers: A sign-off record for each release naming the exact baseline it was measured against, the eval and replay results, and who approved it.
How to build it: Generate the record from the gate's own output rather than writing it by hand after the fact, so it can't drift from what actually ran.
Produces: An answer to "what did we actually check before this shipped" that survives longer than anyone's memory of the release.
Tradeoff: A sign-off record nobody reads is ceremony. It only earns its keep if a real incident review actually pulls it.
Prove it: The sign-off record for the most recent release resolves to the exact gate run it claims, checked by re-running the comparison.

The Eval Stack

03 Progressive exposure

You cannot diff the output of a model change the way you diff a function's return value. A flat percentage rollout is the wrong shape for it. 5% of traffic tells you almost nothing if you only ever look at the error rate. This surface makes exposure gradual. The cohort is chosen because it bounds the damage. A written abort rule replaces a judgment call made mid-incident. A holdback lets you compare against what would have happened anyway. A kill switch has been pulled before the day it matters.

Choose the canary cohort for blast radius, not convenience

Triggered

What it covers: Which users or requests see a change first. Pick them for how much damage a bad version could do. The easiest segment to route to is a poor default.
How to build it: Define the cohort selection rule in advance. Start with low-stakes traffic, a segment where a wrong answer is cheap to correct. Never default to the highest-value accounts.
Produces: A worst case, if the release is bad, that stays small on purpose.
Tradeoff: A canary chosen purely for low stakes can be unrepresentative of the traffic that matters most. A release can look fine on the canary and still fail on a segment it never touched.
Prove it: The current canary cohort's selection rule is written down and was applied, not chosen ad hoc for this release.

Glossary: Canary release

Write the ramp schedule and abort criteria before the first cohort sees it

Triggered

What it covers: A stated sequence of exposure percentages. Each step has a specific, numeric condition that halts the ramp. Decide both before any real traffic sees the change.
How to build it: Write the ramp steps and the abort thresholds into the release record itself. A bad signal during the ramp then triggers a rule. Nobody has to make a judgment call under pressure.
Produces: A ramp that can be paused by whoever is on call, without needing the original author's read on whether a number looks bad.
Tradeoff: A criterion set too tight halts good releases on noise; set too loose, it ramps a bad one past the point of easy recovery. Tune it against real variance, not a guess.
Prove it: The last ramp that hit an abort criterion actually stopped there, checked against the traffic logs, not the postmortem's retelling.

Keep a permanent holdback cohort

Continuous

What it covers: A slice of traffic deliberately kept on the previous release after a new one reaches 100% of everyone else. That gives you a live comparison for as long as it is useful.
How to build it: Carve out a small, fixed cohort at ramp time. It stays on the prior version until the team decides the comparison is no longer needed.
Produces: A real answer to "is the new version actually better" months later, beyond the eval score from the day it shipped.
Tradeoff: Maintaining two live versions is real infrastructure and support cost. Size the holdback to the smallest slice that still gives a statistically useful comparison.
Prove it: The holdback cohort for the current release line still exists and is still receiving traffic, checked directly rather than assumed from the rollout plan.

Exercise the kill switch on a schedule

Periodic

What it covers: Whether the mechanism that instantly reverts exposure to the prior version, or to a safe fallback, has been triggered recently. A description in a runbook is not enough.
How to build it: Run a scheduled drill that pulls the kill switch in a low-stakes window. Confirm that traffic falls back and that tool scopes fall back with it. Do not trust the runbook's description of what the switch does.
Produces: A kill switch that works when someone reaches for it during a real incident, because it worked the last time someone tried it on purpose.
Tradeoff: Drills take engineering time on a fixed schedule, whether or not anything is wrong. A badly timed drill can itself cause a brief disruption.
Prove it: Time to complete the last scheduled kill-switch drill, tracked per drill, not assumed from the runbook.

04 Reversal and pinning

Reverting a normal service means redeploying the last good binary. Reverting an AI system means asking which of five independent things needs to go back. The first is the model, which a provider might have moved out from under you. The others are the prompt, the index the retrieval layer reads, the tool schemas an agent calls and the gating thresholds. Treat any one of those as reversible on its own and the other four keep drifting underneath it. This surface makes "roll back" resolve to one clean, tested action. Nobody has to scramble to work out what state the system was in.

Pin model versions instead of floating to "latest"

Continuous

What it covers: Whether a release names an exact model version, rather than an alias that a provider can silently repoint to a newer model.
How to build it: Reference a specific, dated model version in the release bundle. Treat a provider's alias update as a new release candidate to gate. It is never an automatic upgrade.
Produces: A system whose behavior on a given date is explained by a release you made, not a change a provider made without telling you.
Tradeoff: Pinning means you don't get a provider's quality or safety improvements automatically. Decide the upgrade cadence on purpose instead of inheriting it by default.
Prove it: The model version in production today matches the version named in the last release record. Check it directly against the provider's API. Do not assume it from the config.

Ship prompt, index and tool schemas as one immutable bundle

Continuous

What it covers: A single versioned artifact that pins the exact prompt, retrieval index snapshot and tool schema set live at a given time. The governance layer asks for the same version-as-one-release discipline in its records surface.
How to build it: Bundle the three into one release artifact with one version tag, so reverting the bundle reverts all three together or none of them.
Produces: A rollback that restores an index and a prompt that were built to work together. Nobody reverts a prompt against an index that moved on without it.
Tradeoff: Bundling means a prompt-only fix has to cut a full release instead of a one-line edit. That overhead is what prevents the trap below.
Prove it: The most recent rollback resolved to one bundle version for all three components. Check it against the index and prompt now live. Do not rely on the rollback ticket's account.

Failure mode: Stale index serves deleted content Governance: version a release as one unit

Write the forward-fix-versus-rollback rule before you need it

Triggered

What it covers: A stated rule for when a bad release gets reverted to the last known-good bundle versus patched forward with a fix.
How to build it: Name the deciding factors in advance. They are severity, how far the bad release has ramped, and whether the fix is faster to test than a full rollback. The call then does not start from scratch mid-incident.
Produces: A decision made in minutes during an incident, against a rule everyone already agreed to. The team skips a debate while the bad version keeps serving traffic.
Tradeoff: A rule specific enough to be useful won't cover every case. Treat it as the default that most incidents follow, not a rule that removes judgment entirely.
Prove it: The last release incident's response matches the written rule for its severity and ramp stage, or the deviation is explained in the postmortem.

Track provider deprecation notices on a calendar

Periodic

What it covers: Whether every model, API or library dependency has its known deprecation or end-of-life date recorded where a team checks. The migration is dated before the shutoff.
How to build it: Log each provider's stated deprecation date against the dependency the moment it is announced. Schedule the migration work with a deadline earlier than the shutoff.
Produces: A migration that ships as a planned release with its own gate and rollout. No emergency cutover happens the week a model goes dark.
Tradeoff: Providers change deprecation timelines, and a calendar built on an announced date can still slip. Recheck it, don't treat it as fixed once logged.
Prove it: Every active dependency with a known deprecation date has a migration task scheduled to land before that date. Check it against the current calendar. Do not use the date it was first logged.

Glossary: Model deprecation

05 In-flight watch

A guard metric and a quality metric answer different questions. Conflating them lets a rollout look healthy while it gets worse. P95 latency and tool-error rate tell you the system is still running. They do not tell you it is still right. This surface keeps those two families separate. A human samples real traffic, so the dashboard is not the only check. A named owner is accountable for each drift threshold. None of it works without a trace to read from, which is the piece every other control here depends on.

Track guard metrics separately from quality metrics

Continuous

What it covers: Refusal rate, tool-error rate, p95 latency, and spend per request, monitored as their own dashboard, distinct from whatever measures whether answers are actually correct.
How to build it: Instrument guard metrics at the same trace layer that feeds the eval suite. Alert on each one independently. Do not fold them into one composite health score.
Produces: An early warning for the failures that show up as a number moving, well before a quality regression gets reported by a user.
Tradeoff: More dashboards to watch means more alerts competing for attention. Guard metrics only earn their keep if someone is actually on call for them.
Prove it: The on-call owner for guard metrics can show the last time they acted on one of these alerts specifically. Confirming the dashboard exists doesn't count.

Sample live traffic for human review

Periodic

What it covers: A fixed, recurring sample of real production requests and responses that a person actually reads, rather than trusting aggregate metrics alone to surface a problem.
How to build it: Pull a random, non-trivial sample of live traffic on a fixed cadence. Route it to a named reviewer with a short rubric.
Produces: A caught quality regression that no guard metric would have flagged, because it never threw an error and never spiked latency.
Tradeoff: Human review doesn't scale to every request, and the sample size sets a real limit on what it can catch. It's a complement to metrics, not a replacement.
Prove it: This week's sampled review actually happened and is logged. A calendar entry alone doesn't count.

Set a drift threshold with a named owner

Continuous

What it covers: A defined, numeric threshold for how far a quality or behavior signal can move from its baseline before it counts as drift. A named person is accountable for watching it.
How to build it: Pick a signal that already exists, such as a guard metric or a sampled eval score. Set a threshold against its historical baseline. Name an owner who is expected to act when it is crossed, beyond getting notified.
Produces: A drift event that gets caught while it's still small, instead of discovered three months later when someone finally compares this quarter to last.
Tradeoff: A threshold set too tight fires on normal variance and gets ignored within a month. Calibrate it against real historical noise before trusting it.
Prove it: The named owner can show the specific threshold, the signal it watches, and the last time it was actually crossed.

Glossary: Output drift

Build the trace plumbing every other watch control reads from

Continuous

What it covers: The underlying capture of every prompt, retrieval, tool call and output, tied together by a shared request identifier. Guard metrics, sampled review and drift detection all depend on it.
How to build it: Generate one trace ID per request at the entry point. Propagate it through every downstream call. The Operate stage's trace-first observability pattern names the same discipline.
Produces: A specific, inspectable span for any request someone wants to check, instead of a vague sense that something regressed.
Tradeoff: Full trace capture at this granularity is real storage, and for personal data, a real privacy-scoping decision, not a free add.
Prove it: A request picked at random resolves to a complete, correlated trace across every hop, checked by actually pulling one.

Pattern: Trace-first observability

06 Incident and change record

A traditional outage announces itself: the service is down, the alert fires, everyone knows it's an incident. An AI system's worst failures often look like a normal response, just a wrong one, delivered with the same confidence as a right one. A severity ladder built around crashes and downtime will never catch that. This surface defines severity around being wrong. The first runbook step reduces exposure before anyone diagnoses. Every real incident then becomes a case the eval suiteAn eval suite is a versioned set of test cases that measures whether outputs are good. will catch next time.

Build a severity ladder for a system that fails by being confidently wrong

Triggered

What it covers: A defined severity scale whose top tiers trigger on wrong-but-plausible output reaching a user, the same way they would for downtime or a thrown error.
How to build it: Name the categories explicitly. Examples are a materially wrong answer that reached a user, an unattended action outside its allowed scope and a confirmed injection. Map each to a severity and a response expectation.
Produces: A shared answer to how bad a given failure actually was, instead of a debate about whether it counts as an incident at all.
Tradeoff: A ladder defined too broadly turns routine, low-stakes mistakes into full incident response every time. Calibrate the tiers against what the response actually costs to run.
Prove it: The last three declared incidents each map cleanly to a named tier. Check them against the ladder's own definitions. Do not assign tiers after the fact to fit the response.

The runbook's first step is pin and reduce exposure

Triggered

What it covers: Whether the documented first response to a release incident is to stop the bleeding. That means pinning the last known-good bundle and cutting the ramp before anyone starts root-causing.
How to build it: Write reducing exposure as the literal first numbered step in the runbook, ahead of diagnosis. Drill it on the same schedule as the kill-switch check above.
Produces: A blast radius that stops growing within minutes of a declared incident, independent of how long root cause actually takes.
Tradeoff: Pinning and reducing exposure first means occasionally rolling back something that turns out to have been fine. That's a cheaper mistake than diagnosing first while a bad release keeps ramping.
Prove it: The last incident's timeline shows exposure reduction before root-cause work started, checked against the actual incident log, not the runbook's claim.

Name the disclosure path before an incident forces one

Triggered

What it covers: A defined process for who gets told what, and when, once an incident is severe enough to affect a user or a customer commitment.
How to build it: Pair each severity tier from the ladder above with a stated disclosure obligation. State who is notified internally. State whether an affected user or customer gets told, and by when.
Produces: A disclosure decision made against a stated rule, not improvised for the first time under a deadline someone else is setting.
Tradeoff: Over-disclosing on every low-severity blip erodes trust in the notifications that actually matter. Scope disclosure to the tiers where it's genuinely warranted.
Prove it: The last severity-2-or-higher incident's disclosure matches what the stated rule for that tier requires.

Turn every postmortem into a new eval case

Triggered

What it covers: A rule that a completed incident review must add at least one new, checkable case to the golden set. The case covers the specific input that caused the failure.
How to build it: Make it a required field on the postmortem template. The field holds the eval case id the incident produced. Leave it empty only with a stated reason the incident cannot be tested.
Produces: A gate that gets measurably harder to fool by the same failure twice, instead of a postmortem whose only output is a document nobody re-reads.
Tradeoff: Not every incident reduces cleanly to one test case, and forcing a bad one into the golden set pollutes it. It's fine for a postmortem to name why this incident is the rare exception.
Prove it: Every incident closed in the last quarter resolves to a named eval case, checked against the eval suite's own change history.

The AI Governance Layer

Five ways a release process lies

A green gate is a claim, not a guarantee.#

Every one of these looks like a working release process right up until the release it was supposed to catch.

A green gate on a golden set frozen in time

Looks like: CI shows a passing eval run on every release, month after month, and nobody has touched the golden set in a long while.

Costs you: A green gate stops meaning "this didn't break anything we care about". It starts meaning "this didn't break the specific cases we tested four months ago". That is a much weaker claim wearing the same badge.

Fix: Add a checked freshness expectation on the golden set itself. Treat every closed incident as an obligation to add a case. That is a higher bar than a document to file away.

See the control: A golden-set eval gate blocks a regressing release →

A canary watched for errors only

Looks like: A rollout ramps cleanly through its scheduled steps. The only metric anyone watches is the error rate, and it never moves.

Costs you: A quality regression that never throws an exception rides the ramp all the way to 100% of traffic. It stays invisible to the one metric anyone checks.

Fix: Track guard metrics and quality metrics as two separate families from the start. Gate ramp progression on both. Do not gate on whichever one is easiest to instrument.

See the control: Track guard metrics separately from quality metrics →

A rollback that reverts the code but not the index

Looks like: An incident gets a fast, confident rollback: the previous prompt and model version are back within minutes.

Costs you: The index kept rebuilding on its own schedule the whole time. The "reverted" system now runs a prompt built against one index while it reads from a different one. The answers are wrong in a new way nobody planned for.

Fix: Bundle prompt, index and tool schemas as one versioned unit. Make reverting the bundle the only rollback path. Then no one can revert one piece without the others.

See the control: Ship prompt, index and tool schemas as one immutable bundle →

A kill switch nobody has pulled since the day it was written

Looks like: The runbook names a clear, one-step way to cut exposure instantly, and it reads as reassuring right up until someone actually needs it.

Costs you: A kill switch that has never been exercised is a hypothesis about what would happen. It is not a tested capability. The incident that needs it is the worst moment to learn the hypothesis was wrong.

Fix: Put the kill switch on the same scheduled-drill discipline as a backup restore test, and log the result every time. The first drill proves nothing about the tenth.

See the control: Exercise the kill switch on a schedule →

"Zero incidents" because nothing was measured well enough to call one

Looks like: A quarterly report proudly states zero AI incidents, and the team genuinely believes it, because nothing tripped the alerting they have in place.

Costs you: A system can fail by being confidently wrong instead of crashing. Under a severity ladder built for outages, it can run that way for a long time. A real absence of harm and a fake absence of failures look identical from the dashboard.

Fix: Build the severity ladder around wrong-but-plausible output. Pair it with sampled human review that surfaces the failures a metric alone would miss.

See the control: Build a severity ladder for a system that fails by being confidently wrong →

The scoreboard

Six numbers, and how each one lies.#

Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.

Change lead time

Why it matters: Whether the release path is fast enough that people actually use it, instead of finding a way to route a change around the gate entirely.
How it misleads: A fast lead time can just as easily describe a gate that lets everything through unexamined as a genuinely efficient one.
Pair with: Share of changes that actually cleared the golden-set gate rather than shipping on an override

Canary-to-full ramp duration

Why it matters: The direct measure of whether exposure is genuinely progressive or a percentage label on what was really one big flip.
How it misleads: A long duration on paper can still hide a single jump from a small canary straight to full traffic. Nothing is observed at the steps in between.
Pair with: Number of distinct ramp steps actually observed before reaching full exposure

Reversal MTTR

Why it matters: Whether "we can roll back" is a tested capability or an assertion nobody has had to rely on yet.
How it misleads: A fast MTTR on a code-only rollback says nothing about whether the index and prompt bundle it depends on actually reverted with it.
Pair with: Share of rollbacks that also reverted the index and prompt bundle, not code alone

Guard-metric alert precision

Why it matters: Whether the team trusts a guard-metric alert enough to act on it during a live ramp, instead of treating it as noise to dismiss.
How it misleads: High precision on a narrow set of guard metrics can still hide zero coverage on the specific signal that actually caught the last real incident.
Pair with: Guard-metric alert recall: the share of real regressions the metrics actually caught

Share of releases with a tested reversal path

Why it matters: Distinguishes a rollback that has been drilled from one that has only ever been described in a runbook.
How it misleads: A high share can still hide a drill run against a synthetic scenario. It tests the mechanism without proving it against a genuine failure on a real past release.
Pair with: Time since the reversal path was last exercised against an actual past release

Drift-detection lag

Why it matters: The real test of whether a drift threshold is a working detector or a metric with nobody watching it.
How it misleads: A short lag on a well-instrumented flagship feature says nothing about a quieter feature sharing the same model without the same dashboard.
Pair with: Number of distinct systems the drift threshold is actually applied to, beyond the one feature it was originally built for

What this is built from

Verifiable, not claimed.#

No invented reliability stats. Just what's already documented on this site, and how it connects.

The eval stack is what a golden-set gate actually runs. This page picks up right after a change clears it: how it reaches production gradually, and how it comes back if it doesn't. The Eval Stack
Governance's monitoring and change-control surface is the standing regime a release has to report into. It says who watches the release, what counts as an incident and what re-triggers a review. This page is the release mechanics underneath that regime. The AI Governance Layer
The Agentic Delivery Playbook covers how a change gets built and merged with a fleet of coding agents. This page starts once that change is ready to ship. The Agentic Delivery Playbook
The Delivery Record publishes six real defects that shipped on this codebase and the build-time check each one left behind. It's the same records-and-traceability argument this page makes, evidenced from our own history instead of asserted. The Delivery Record
The golden-set gate pattern is the CI mechanics behind the gate surface: what turns "we have an eval set" into "a regression can't merge." Pattern: Golden-set gate in CI
Trace-first observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. is the infrastructure every watch control on this page depends on. It uses one trace ID per request, propagated through every hop. A regression then becomes a specific span. Pattern: Trace-first observability

Sources

Google DORA - DORA Research
The research programme behind the delivery metrics our release and delivery pages measure. Retrieved 2026-08-24.
Google - Implementing SLOs
The method for setting a service-level objective our release scoreboard follows. Retrieved 2026-08-24.
Google - Error Budget Policy
The error-budget policy pattern our release gate borrows. Retrieved 2026-08-24.

Not sure your rollback would work?

A Ship Audit checks this release path against your system. It tells you which control is real and which one has never been exercised.

Book a Ship Audit → See the eval stack →

Questions

Before you ship the next change.#

What teams ask us before they build a real release process for an AI system.

01 Isn't this just CI/CD with different names?

Mechanically, much of it looks like ordinary CI/CD: a gate, a ramp, a rollback. The difference is that you can't diff the output of a model change the way you diff a function's return value. The gate has to run an eval suite instead of a test suite. Reversal has to account for an index and a prompt that drift independently of the code.

Link to this answer

02 We just edit the prompt in a dashboard. Do we really need a release process for that?

If the prompt change can reach real users without going through a gate or a rollback path, then yes, treat it as a release. A bad prompt edit has the same blast radius as a bad code deploy. The process is usually the missing piece. The risk is already there.

Link to this answer

03 What's the difference between this and the Operate stage in the handbook?

Operate is about detecting a failure once a system is already live: unbounded loops, silent tool failures, a stale index. This page covers the mechanics of shipping the change in the first place. It shows how a change gets gated, ramped and reversed. Fewer of those failures then reach 100% of production before anyone notices.

Link to this answer

04 How do you roll back a retrieval index that already got rebuilt?

You don't roll back a rebuild in place. You version the index build itself, the way you would version a model. Then you pin the previous build as the fallback. That is the whole argument for shipping prompt, index and tool schemas as one bundle. Reverting means switching back to a previous, complete bundle. You never try to undo a rebuild after the fact.

Link to this answer

05 What counts as a severity-1 release incident if the system never actually crashes?

A harmful or materially wrong answer reaching a real user, at scale or in a high-stakes context, counts the same as an outage would. The severity ladder here starts from confidently wrong output rather than from downtime, because that's the failure mode an AI system is actually prone to.

Link to this answer

06 Do we need all six surfaces before we ship anything?

No. A golden-set gate and a documented rollback path cost little and catch most of the risk on a first release. Scale the rest to the exposure and the number of independent moving parts the system has. The rest means the canary cohort discipline, the drift thresholds and the incident ladder. The moving parts are the model, prompt, index and tools.

Link to this answer

Operate what you just shipped

A rollback that leaves the index untouched is not a real rollback, and an unpulled kill switch is not a safeguard. The Operate stage covers what a release needs after it ships.

Read the Operate stage → Talk to us →