# The AI Release Path: ship a change, ramp it, and reverse it if it's wrong.

Source: https://customlabs.io/release/
Updated: 2026-09-20

Release

# The AI Release Path: ship a change, ramp it, and reverse it if it's wrong.

Six surfaces, 24 named controls, five ways a release process lies, and the scoreboard that keeps them honest. A code release has an obvious boundary; an AI system's behavior can change without a line of code moving, and this is the discipline for shipping that change on purpose.

Updated September 20, 2026 · First published August 13, 2026

·

34 min read

·

Key takeaways

- → A rollback that reverts the code but leaves the index untouched isn't a real rollback.
- → A kill switch nobody has pulled since it was written is a named trap, not a safeguard.
- → Reversal MTTR measures how fast a bad release actually gets undone.
- → Progressive exposure and 'reversal and pinning' are two separate named surfaces.
- → Guard-metric alert precision catches an alert that fires on noise instead of a real regression.
- → 24 controls span six surfaces, from what counts as a release to the incident record.

You can't diff the output of a model change the way you diff a function's return value, which is why a flat percentage rollout and a git revert both fall short here. The model, the prompt, the retrieval index, the tool schemas, and the thresholds gating all of it are five independent moving parts, and any one of them can change behavior on its own. This page is the discipline for treating that seriously: what counts as a release, how it earns exposure, and what "revert" actually has to mean once the index was rebuilt in place.

The Handbook's [Operate stage](https://customlabs.io/handbook/operate/) covers detecting a failure once a system is live. This page is what happens before that: how a change actually ships, so fewer failures reach production ramped to 100% before anyone notices.

The six surfaces

## What a release process actually has to cover.

Each surface fails a different way. None of them substitute for each other.

A release moves through six stages in order, from what counts as a release to the incident record. A reversal can trigger a new release cycle.

01

### What counts as a release

**Covers:** Every input that changes system behavior: model version, system prompt, retrieval corpus and index build, tool set and schemas, thresholds and config, and the harness or orchestration code around all of it.

**Breaks when:** An index gets rebuilt in place on a cron job nobody calls a deploy, so a behavior change ships with none of the review, the gate, or the rollback plan an actual release would get.

**Watch:** Share of behavior-changing inputs treated as releases

02

### The gate before exposure

**Covers:** The checks a change has to clear before any real user sees it: a golden-set eval gate in CI, an offline replay against recorded production traffic, a cost and latency budget check, and a sign-off record naming what was actually compared against what.

**Breaks when:** The gate is green, and the golden set it ran against hasn't changed in months, so a green light means the change didn't break the cases everyone already agreed it wouldn't.

**Watch:** Golden-set freshness (days since a new case was added)

03

### Progressive exposure

**Covers:** How a change reaches real users gradually instead of all at once: a canary cohort picked for blast radius, a ramp schedule with abort criteria written down first, a permanent holdback for comparison, and a kill switch that actually gets pulled on a schedule.

**Breaks when:** A canary is watched only for hard errors, so a quality regression that never throws an exception rides the ramp all the way to 100% of traffic before anyone notices the answers got worse.

**Watch:** Canary-to-full ramp duration

04

### Reversal and pinning

**Covers:** What "revert" actually has to mean once the model, the prompt, the index and the tool schemas can each change independently: pinned model versions, one immutable bundle per release, a stated rule for forward-fix versus rollback, and a provider deprecation calendar with the migration dated before the shutoff.

**Breaks when:** A rollback restores last week's code but not the index it was built against, so the "reverted" system answers from an index that no longer matches the prompt reading it.

**Watch:** Reversal MTTR

05

### In-flight watch

**Covers:** What gets monitored once a change is live: guard metrics (refusal rate, tool-error rate, p95 latency, spend per request) tracked separately from quality metrics, sampled human review of live traffic, drift thresholds with a named owner, and the trace plumbing all of it actually reads from.

**Breaks when:** A dashboard tracks refusal rate and latency, both look fine, and the system has been quietly answering worse questions worse for two weeks because nothing on that dashboard measures correctness.

**Watch:** Drift-detection lag

06

### Incident and change record

**Covers:** What happens once something goes wrong: a severity ladder built for a system that fails by being confidently wrong rather than by crashing, a runbook whose first step is to pin and reduce exposure, a stated disclosure path, and a rule that every postmortem becomes a new eval case.

**Breaks when:** A team reports "zero incidents" for the quarter, and the honest reading is that nothing was measured well enough for anyone to recognize one when it happened.

**Watch:** Declared incidents with a resulting eval case

The control bank

## 24 controls, six surfaces.

Filter by how often it's exercised (continuous, periodic, triggered), then copy the visible list as a Markdown checklist.

Continuous Periodic Triggered

Showing all 24 controls

Copy as Markdown

### 01 What counts as a release

A code release has an obvious boundary: a commit merges, a build runs, a version ships. An AI system's behavior can change without a single line of code moving: a provider quietly updates a model behind an alias, a nightly job rebuilds the retrieval index, someone edits a threshold in a config panel. None of those look like a release, and none of them get a release's scrutiny, which is exactly why this surface exists: name every input that can change what the system does, on purpose, before deciding how any of it ships.

#### Name every input that counts as a release

Triggered

**What it covers**
An explicit, written list of everything that changes system behavior: model version, system prompt, retrieval index build, tool set and schemas, thresholds and config, and the harness code coordinating them.

**How to build it**
Write the list once, per system, and treat any change to an item on it as a release, whatever team or process actually makes the change.

**Produces**
A shared answer to "did anything change" that doesn't depend on remembering every team that can touch behavior.

**Tradeoff**
A list this broad pulls config and index changes under process that used to move without any review at all. That friction is the point, not a bug in the list.

**Prove it**
Every behavior-changing input on the list has a release record attached to its last change, checked against the actual change history, not the list's own claim.

#### Treat an index rebuild as a release, not a maintenance job

Periodic

**What it covers**
Whether rebuilding the retrieval index, on a schedule or on demand, goes through the same release discipline as a model or prompt change.

**How to build it**
Route index rebuilds through the same gate and versioning as any other release input, instead of a cron job that runs unattended and unreviewed.

**Produces**
A rebuilt index that carries a version, a gate result, and a rollback path, the same as a code deploy.

**Tradeoff**
Gating every rebuild adds latency to what used to be a fire-and-forget job. Size the gate to the rebuild's actual blast radius rather than blocking a routine refresh on a full eval run.

**Prove it**
The last three index rebuilds each resolve to a release record, checked against the ingest job's own logs.

#### Treat a threshold or config change as a release

Continuous

**What it covers**
A rule that a changed confidence threshold, routing rule, or guardrail setting goes through the same review a prompt change would, rather than a config panel edit nobody logs.

**How to build it**
Route config and threshold edits through source control or an equivalent change log, with the same reviewer expectations as a code change.

**Produces**
A change history for the numbers that quietly decide how the system behaves, the same discipline the code around them already gets.

**Tradeoff**
This is real friction on values teams are used to tuning live. The alternative is a system whose actual behavior at any past date can't be reconstructed.

**Prove it**
Every threshold currently in production traces to a dated, reviewed change record.

[Glossary: Model deprecation](https://customlabs.io/glossary/model-deprecation/)

#### Give every release one named owner

Continuous

**What it covers**
A single person accountable for a given release, distinct from whoever happened to trigger the underlying change.

**How to build it**
A required owner field on the release record, assigned before the change goes out, not reconstructed afterward from commit history.

**Produces**
A single name to ask when a release does something nobody expected.

**Tradeoff**
Ownership needs to survive a reorg or a departure. An owner field pointing at someone who left is worse than an honest blank.

**Prove it**
A release record picked at random names a current, reachable owner.

### 02 The gate before exposure

A code review asks whether a diff is correct. An AI release gate has to ask a harder question: does this change make the system worse on inputs nobody wrote a test for. The eval stack answers that for known cases; this surface is what turns a passing eval run into a real go/no-go decision, backed by a record of what was compared, replayed against traffic the system actually saw, and checked against a budget before the invoice makes the case for you.

#### A golden-set eval gate blocks a regressing release

Continuous

**What it covers**
A CI check that runs the labelled eval set against a candidate release and blocks it from merging or shipping if the pass rate drops.

**How to build it**
Wire the eval suite into CI as a required check, the same way a unit test suite gates a code change, with an explicit, reviewed override for a deliberate trade-off.

**Produces**
A release that can't reach production carrying a known regression nobody signed off on.

**Tradeoff**
A gate this strict blocks a release on a case that later turns out not to matter. That is a cheaper failure mode than shipping a regression the gate could have caught.

**Prove it**
The last ten merges to the release branch each show a passing, logged gate run, not an override with no reason attached.

[Pattern: Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/)

#### Replay recorded production traffic before shipping

Periodic

**What it covers**
Running a candidate release against a sample of real, previously captured production requests, offline, before it ever serves a real user.

**How to build it**
Capture a rolling sample of production inputs (with the same privacy handling live traffic gets) and replay it against every candidate release as a required gate step.

**Produces**
Evidence about how a change behaves on the actual input distribution, a wider check than the golden set someone assembled once.

**Tradeoff**
A useful replay sample has to be refreshed as real traffic shifts, or it becomes its own stale golden set with an extra step.

**Prove it**
The replay sample was refreshed inside the last release cycle, checked against its own capture date.

[Glossary: Shadow deployment](https://customlabs.io/glossary/shadow-deployment/)

#### Check the cost and latency budget before exposure, not after the invoice

Continuous

**What it covers**
A gate step that measures a candidate release's cost per request and p95 latency against a stated budget, before it reaches any real traffic.

**How to build it**
Run the same replay sample used for the quality gate through a cost and latency measurement, and block a release that clears its budget by too wide a margin.

**Produces**
A release that can't ship a quiet 3x cost regression that would otherwise only show up on next month's bill.

**Tradeoff**
A budget set once and never revisited against real usage becomes exactly the stale check it was meant to replace.

**Prove it**
The budget for the current release line was checked against last month's actual spend, not the original launch estimate.

#### Record what was actually compared against what

Triggered

**What it covers**
A sign-off record for each release naming the exact baseline it was measured against, the eval and replay results, and who approved it.

**How to build it**
Generate the record from the gate's own output rather than writing it by hand after the fact, so it can't drift from what actually ran.

**Produces**
An answer to "what did we actually check before this shipped" that survives longer than anyone's memory of the release.

**Tradeoff**
A sign-off record nobody reads is ceremony. It only earns its keep if a real incident review actually pulls it.

**Prove it**
The sign-off record for the most recent release resolves to the exact gate run it claims, checked by re-running the comparison.

[The Eval Stack](https://customlabs.io/evals/)

### 03 Progressive exposure

You cannot diff the output of a model change the way you diff a function's return value, which is exactly why a flat percentage rollout is the wrong shape for it: 5% of traffic tells you almost nothing if you were only ever going to look at the error rate. This surface is what makes exposure genuinely gradual: a cohort chosen because it bounds the damage, a written abort rule instead of a judgment call made mid-incident, a holdback that lets you compare against what would have happened anyway, and a kill switch that has actually been pulled before the day it matters.

#### Choose the canary cohort for blast radius, not convenience

Triggered

**What it covers**
Which users or requests see a change first, picked deliberately for how much damage a bad version could do, not for whichever segment is easiest to route to.

**How to build it**
Define the cohort selection rule in advance: low-stakes traffic first, a segment where a wrong answer is cheap to correct, never the highest-value accounts by default.

**Produces**
A worst case, if the release is bad, that stays small on purpose.

**Tradeoff**
A canary chosen purely for low stakes can also be unrepresentative of the traffic that matters most, and a release can look fine on the canary and still fail on the segment it never touched.

**Prove it**
The current canary cohort's selection rule is written down and was applied, not chosen ad hoc for this release.

[Glossary: Canary release](https://customlabs.io/glossary/canary-release/)

#### Write the ramp schedule and abort criteria before the first cohort sees it

Triggered

**What it covers**
A stated sequence of exposure percentages and the specific, numeric condition that halts the ramp at each step, decided before any real traffic sees the change.

**How to build it**
Write the ramp steps and the abort thresholds into the release record itself, so a bad signal during the ramp triggers a rule instead of a judgment call made under pressure.

**Produces**
A ramp that can be paused by whoever is on call, without needing the original author's read on whether a number looks bad.

**Tradeoff**
A criterion set too tight halts good releases on noise; set too loose, it ramps a bad one past the point of easy recovery. Tune it against real variance, not a guess.

**Prove it**
The last ramp that hit an abort criterion actually stopped there, checked against the traffic logs, not the postmortem's retelling.

#### Keep a permanent holdback cohort

Continuous

**What it covers**
A slice of traffic deliberately kept on the previous release even after a new one reaches 100% of everyone else, so there's a live comparison for as long as it's useful.

**How to build it**
Carve out a small, fixed cohort at ramp time that stays on the prior version until the team explicitly decides the comparison is no longer needed.

**Produces**
A real answer to "is the new version actually better" months later, beyond the eval score from the day it shipped.

**Tradeoff**
Maintaining two live versions is real infrastructure and support cost. Size the holdback to the smallest slice that still gives a statistically useful comparison.

**Prove it**
The holdback cohort for the current release line still exists and is still receiving traffic, checked directly rather than assumed from the rollout plan.

#### Exercise the kill switch on a schedule

Periodic

**What it covers**
Whether the mechanism that instantly reverts exposure to the prior version, or to a safe fallback, has actually been triggered recently, beyond having a description in a runbook.

**How to build it**
Run a scheduled drill that pulls the kill switch in a low-stakes window and confirms traffic actually falls back, tool scopes included, rather than trusting the runbook's description of what it does.

**Produces**
A kill switch that works when someone reaches for it during a real incident, because it worked the last time someone tried it on purpose.

**Tradeoff**
Drills take engineering time on a fixed schedule whether or not anything is currently wrong, and a badly timed drill can itself cause a brief disruption.

**Prove it**
Time to complete the last scheduled kill-switch drill, tracked per drill, not assumed from the runbook.

### 04 Reversal and pinning

Reverting a normal service means redeploying the last good binary. Reverting an AI system means asking which of five independent things actually needs to go back: the model, which a provider might have moved out from under you; the prompt; the index the retrieval layer reads; the tool schemas an agent calls against; the thresholds gating all of it. Treat any one of those as reversible on its own and the other four keep drifting underneath it. This surface is the discipline that makes "roll back" resolve to one clean, tested action instead of a scramble to figure out what state the system was actually in.

#### Pin model versions instead of floating to "latest"

Continuous

**What it covers**
Whether a release names an exact model version, rather than an alias that a provider can silently repoint to a newer model.

**How to build it**
Reference a specific, dated model version in the release bundle, and treat a provider's alias update as a new release candidate to gate, not an automatic upgrade.

**Produces**
A system whose behavior on a given date is explained by a release you made, not a change a provider made without telling you.

**Tradeoff**
Pinning means you don't get a provider's quality or safety improvements automatically. Decide the upgrade cadence on purpose instead of inheriting it by default.

**Prove it**
The model version in production today matches the version named in the last release record, checked directly against the provider's API, not assumed from the config.

#### Ship prompt, index and tool schemas as one immutable bundle

Continuous

**What it covers**
A single versioned artifact that pins the exact prompt, retrieval index snapshot, and tool schema set live at a given time, the same version-as-one-release discipline the governance layer asks for in its records surface.

**How to build it**
Bundle the three into one release artifact with one version tag, so reverting the bundle reverts all three together or none of them.

**Produces**
A rollback that restores an index and a prompt that were actually built to work together, instead of a prompt reverted against an index that moved on without it.

**Tradeoff**
Bundling means a prompt-only fix has to cut a full release instead of a one-line edit. That overhead is what prevents the trap below.

**Prove it**
The most recent rollback resolved to one bundle version for all three components, checked against the actual index and prompt now live, not the rollback ticket's description of what happened.

[Failure mode: Stale index serves deleted content](https://customlabs.io/failure-modes/stale-index-serves-deleted-content/)[Governance: version a release as one unit](https://customlabs.io/governance/#version-release-as-one-unit)

#### Write the forward-fix-versus-rollback rule before you need it

Triggered

**What it covers**
A stated rule for when a bad release gets reverted to the last known-good bundle versus patched forward with a fix.

**How to build it**
Name the deciding factors in advance: severity, how far the bad release has ramped, and whether the fix is faster to test than a full rollback, so the call doesn't get made from scratch mid-incident.

**Produces**
A decision made in minutes during an incident, against a rule everyone already agreed to, instead of a debate while the bad version keeps serving traffic.

**Tradeoff**
A rule specific enough to be useful won't cover every case. Treat it as the default that most incidents follow, not a rule that removes judgment entirely.

**Prove it**
The last release incident's response matches the written rule for its severity and ramp stage, or the deviation is explained in the postmortem.

#### Track provider deprecation notices on a calendar

Periodic

**What it covers**
Whether every model, API, or library dependency's known deprecation or end-of-life date is recorded somewhere a team actually checks, with the migration dated before the shutoff.

**How to build it**
Log each provider's stated deprecation date against the dependency the moment it's announced, and schedule the migration work with a deadline earlier than the shutoff, not on it.

**Produces**
A migration that ships as a planned release with its own gate and rollout, instead of an emergency cutover the week a model actually goes dark.

**Tradeoff**
Providers change deprecation timelines, and a calendar built on an announced date can still slip. Recheck it, don't treat it as fixed once logged.

**Prove it**
Every active dependency with a known deprecation date has a migration task scheduled to land before that date, checked against the current calendar, not the date it was first logged.

[Glossary: Model deprecation](https://customlabs.io/glossary/model-deprecation/)

### 05 In-flight watch

A guard metric and a quality metric answer different questions, and conflating them is how a rollout looks healthy while it's actively getting worse: p95 latency and tool-error rate tell you the system is still running, not that it's still right. This surface keeps those two families separate, adds a human in the loop sampling real traffic instead of trusting the dashboard alone, and puts a named owner on a drift threshold instead of a metric nobody is actually accountable for. None of it works without a trace to read from, which is the piece every other control here depends on.

#### Track guard metrics separately from quality metrics

Continuous

**What it covers**
Refusal rate, tool-error rate, p95 latency, and spend per request, monitored as their own dashboard, distinct from whatever measures whether answers are actually correct.

**How to build it**
Instrument guard metrics at the same trace layer that feeds the eval suite, and alert on them independently instead of folding them into one composite health score.

**Produces**
An early warning for the failures that show up as a number moving, well before a quality regression gets reported by a user.

**Tradeoff**
More dashboards to watch means more alerts competing for attention. Guard metrics only earn their keep if someone is actually on call for them.

**Prove it**
The on-call owner for guard metrics can show the last time they acted on one of these alerts specifically. Confirming the dashboard exists doesn't count.

#### Sample live traffic for human review

Periodic

**What it covers**
A fixed, recurring sample of real production requests and responses that a person actually reads, rather than trusting aggregate metrics alone to surface a problem.

**How to build it**
Pull a random, non-trivial sample of live traffic on a fixed cadence and route it to a named reviewer with a short rubric to check against.

**Produces**
A caught quality regression that no guard metric would have flagged, because it never threw an error and never spiked latency.

**Tradeoff**
Human review doesn't scale to every request, and the sample size sets a real limit on what it can catch. It's a complement to metrics, not a replacement.

**Prove it**
This week's sampled review actually happened and is logged. A calendar entry alone doesn't count.

#### Set a drift threshold with a named owner

Continuous

**What it covers**
A defined, numeric threshold for how far a quality or behavior signal can move from its baseline before it counts as drift, plus a named person accountable for watching it.

**How to build it**
Pick a signal that already exists (a guard metric, a sampled eval score) and set a threshold against its historical baseline, with an owner who is expected to act when it's crossed, past simply getting notified.

**Produces**
A drift event that gets caught while it's still small, instead of discovered three months later when someone finally compares this quarter to last.

**Tradeoff**
A threshold set too tight fires on normal variance and gets ignored within a month. Calibrate it against real historical noise before trusting it.

**Prove it**
The named owner can show the specific threshold, the signal it watches, and the last time it was actually crossed.

[Glossary: Output drift](https://customlabs.io/glossary/output-drift/)

#### Build the trace plumbing every other watch control reads from

Continuous

**What it covers**
The underlying capture of every prompt, retrieval, tool call and output, tied together by a shared request identifier, that guard metrics, sampled review and drift detection all depend on.

**How to build it**
Generate one trace ID per request at the entry point and propagate it through every downstream call, the same discipline the Operate stage's trace-first observability pattern names directly.

**Produces**
A specific, inspectable span for any request someone wants to check, instead of a vague sense that something regressed.

**Tradeoff**
Full trace capture at this granularity is real storage, and for personal data, a real privacy-scoping decision, not a free add.

**Prove it**
A request picked at random resolves to a complete, correlated trace across every hop, checked by actually pulling one.

[Pattern: Trace-first observability](https://customlabs.io/patterns/trace-first-observability/)

### 06 Incident and change record

A traditional outage announces itself: the service is down, the alert fires, everyone knows it's an incident. An AI system's worst failures often look like a normal response, just a wrong one, delivered with the same confidence as a right one. A severity ladder built around crashes and downtime will never catch that, which is why this surface starts by defining severity around being wrong instead of being unavailable, then makes the first runbook step reduce exposure rather than diagnose, and closes the loop by turning every real incident into a case the [eval suite](https://customlabs.io/glossary/eval-suite/) will catch next time.

#### Build a severity ladder for a system that fails by being confidently wrong

Triggered

**What it covers**
A defined severity scale whose top tiers trigger on wrong-but-plausible output reaching a user, the same way they would for downtime or a thrown error.

**How to build it**
Name the categories explicitly: a harmful or materially wrong answer that reached a user, an unattended action outside its allowed scope, a confirmed injection, each mapped to a severity and a response expectation.

**Produces**
A shared answer to how bad a given failure actually was, instead of a debate about whether it counts as an incident at all.

**Tradeoff**
A ladder defined too broadly turns routine, low-stakes mistakes into full incident response every time. Calibrate the tiers against what the response actually costs to run.

**Prove it**
The last three declared incidents each map cleanly to a named tier, checked against the ladder's own definitions, not assigned after the fact to fit however the response went.

#### The runbook's first step is pin and reduce exposure

Triggered

**What it covers**
Whether the documented first response to a release incident is to stop the bleeding (pin the last known-good bundle, cut the ramp) before anyone starts root-causing.

**How to build it**
Write reducing exposure as the literal first numbered step in the runbook, ahead of diagnosis, and drill it on the same schedule as the kill-switch check above.

**Produces**
A blast radius that stops growing within minutes of a declared incident, independent of how long root cause actually takes.

**Tradeoff**
Pinning and reducing exposure first means occasionally rolling back something that turns out to have been fine. That's a cheaper mistake than diagnosing first while a bad release keeps ramping.

**Prove it**
The last incident's timeline shows exposure reduction before root-cause work started, checked against the actual incident log, not the runbook's claim.

#### Name the disclosure path before an incident forces one

Triggered

**What it covers**
A defined process for who gets told what, and when, once an incident is severe enough to affect a user or a customer commitment.

**How to build it**
Pair each severity tier from the ladder above with a stated disclosure obligation: who's notified internally, and whether an affected user or customer gets told, and by when.

**Produces**
A disclosure decision made against a stated rule, not improvised for the first time under a deadline someone else is setting.

**Tradeoff**
Over-disclosing on every low-severity blip erodes trust in the notifications that actually matter. Scope disclosure to the tiers where it's genuinely warranted.

**Prove it**
The last severity-2-or-higher incident's disclosure matches what the stated rule for that tier requires.

#### Turn every postmortem into a new eval case

Triggered

**What it covers**
A rule that a completed incident review must add at least one new, checkable case to the golden set, covering the specific input that caused the failure.

**How to build it**
Make it a required field on the postmortem template: the eval case id the incident produced, rejected only with a stated reason it genuinely can't be tested for.

**Produces**
A gate that gets measurably harder to fool by the same failure twice, instead of a postmortem whose only output is a document nobody re-reads.

**Tradeoff**
Not every incident reduces cleanly to one test case, and forcing a bad one into the golden set pollutes it. It's fine for a postmortem to name why this incident is the rare exception.

**Prove it**
Every incident closed in the last quarter resolves to a named eval case, checked against the eval suite's own change history.

[The AI Governance Layer](https://customlabs.io/governance/)

No controls match that combination. Clear a filter to see more.

Five ways a release process lies

## A green gate is a claim, not a guarantee.

Every one of these looks like a working release process right up until the release it was supposed to catch.

### A green gate on a golden set frozen in time

**Looks like:** CI shows a passing eval run on every release, month after month, and nobody has touched the golden set in a long while.

**Costs you:** A green gate stops meaning "this didn't break anything we care about" and starts meaning "this didn't break the specific cases we tested four months ago," which is a much weaker claim wearing the same badge.

**Fix:** Add a real, checked freshness expectation on the golden set itself, and treat every closed incident as an obligation to add a case, a higher bar than a document to file away.

[See the control: A golden-set eval gate blocks a regressing release →](https://customlabs.io/release/#golden-set-gate-blocks-regression)

### A canary watched for errors only

**Looks like:** A rollout ramps cleanly through its scheduled steps because the only thing anyone is watching is the error rate, and the error rate never moves.

**Costs you:** A quality regression that never throws an exception rides the ramp all the way to 100% of traffic, fully invisible to the one metric anyone was actually checking.

**Fix:** Track guard metrics and quality metrics as two separate families from the start, and gate ramp progression on both, not on whichever one is easiest to instrument.

[See the control: Track guard metrics separately from quality metrics →](https://customlabs.io/release/#track-guard-metrics-separately)

### A rollback that reverts the code but not the index

**Looks like:** An incident gets a fast, confident rollback: the previous prompt and model version are back within minutes.

**Costs you:** The index kept rebuilding on its own schedule the whole time, so the "reverted" system is now running a prompt built against one index, reading from a different one, and the answers are wrong in a new way nobody planned for.

**Fix:** Bundle prompt, index and tool schemas as one versioned unit, and make reverting the bundle the only rollback path, so there is no way to revert one piece without the others.

[See the control: Ship prompt, index and tool schemas as one immutable bundle →](https://customlabs.io/release/#ship-prompt-index-tools-as-one-bundle)

### A kill switch nobody has pulled since the day it was written

**Looks like:** The runbook names a clear, one-step way to cut exposure instantly, and it reads as reassuring right up until someone actually needs it.

**Costs you:** A kill switch that has never been exercised is a hypothesis about what would happen, not a tested capability, and the incident it's needed for is the worst possible moment to discover the hypothesis was wrong.

**Fix:** Put the kill switch on the same scheduled-drill discipline as a backup restore test, and log the result every time. The first drill proves nothing about the tenth.

[See the control: Exercise the kill switch on a schedule →](https://customlabs.io/release/#exercise-the-kill-switch-on-a-schedule)

### "Zero incidents" because nothing was measured well enough to call one

**Looks like:** A quarterly report proudly states zero AI incidents, and the team genuinely believes it, because nothing tripped the alerting they have in place.

**Costs you:** A system that fails by being confidently wrong instead of by crashing can run that way for a long time under a severity ladder built for outages, producing a real absence of harm and a fake absence of failures that look identical from the dashboard.

**Fix:** Build the severity ladder around wrong-but-plausible output specifically, and pair it with sampled human review that would actually surface the failure a metric alone would miss.

[See the control: Build a severity ladder for a system that fails by being confidently wrong →](https://customlabs.io/release/#build-a-severity-ladder-for-confidently-wrong)

The scoreboard

## Six numbers, and how each one lies.

Every one of these is measurable today. None of them is trustworthy read alone. Pair it with the number next to it.

### Change lead time

**Why it matters**
Whether the release path is fast enough that people actually use it, instead of finding a way to route a change around the gate entirely.

**How it misleads**
A fast lead time can just as easily describe a gate that lets everything through unexamined as a genuinely efficient one.

**Pair with**
Share of changes that actually cleared the golden-set gate rather than shipping on an override

### Canary-to-full ramp duration

**Why it matters**
The direct measure of whether exposure is genuinely progressive or a percentage label on what was really one big flip.

**How it misleads**
A long duration on paper can still hide a single jump from a small canary straight to full traffic, with nothing observed at the steps in between.

**Pair with**
Number of distinct ramp steps actually observed before reaching full exposure

### Reversal MTTR

**Why it matters**
Whether "we can roll back" is a tested capability or an assertion nobody has had to rely on yet.

**How it misleads**
A fast MTTR on a code-only rollback says nothing about whether the index and prompt bundle it depends on actually reverted with it.

**Pair with**
Share of rollbacks that also reverted the index and prompt bundle, not code alone

### Guard-metric alert precision

**Why it matters**
Whether the team trusts a guard-metric alert enough to act on it during a live ramp, instead of treating it as noise to dismiss.

**How it misleads**
High precision on a narrow set of guard metrics can still hide zero coverage on the specific signal that actually caught the last real incident.

**Pair with**
Guard-metric alert recall: the share of real regressions the metrics actually caught

### Share of releases with a tested reversal path

**Why it matters**
Distinguishes a rollback that has been drilled from one that has only ever been described in a runbook.

**How it misleads**
A high share can still hide a drill run against a synthetic scenario rather than a real past release, which tests the mechanism without proving it against a genuine failure.

**Pair with**
Time since the reversal path was last exercised against an actual past release

### Drift-detection lag

**Why it matters**
The real test of whether a drift threshold is a working detector or a metric with nobody watching it.

**How it misleads**
A short lag on a well-instrumented flagship feature says nothing about a quieter feature sharing the same model without the same dashboard.

**Pair with**
Number of distinct systems the drift threshold is actually applied to, beyond the one feature it was originally built for

What this is built from

## Verifiable, not claimed.

No invented reliability stats. Just what's already documented on this site, and how it connects.

- The eval stack is what a golden-set gate actually runs. This page picks up right after a change clears it: how it reaches production gradually, and how it comes back if it doesn't. [The Eval Stack](https://customlabs.io/evals/)
- Governance's monitoring and change-control surface is the standing regime a release has to report into: who watches it, what counts as an incident, what re-triggers a review. This page is the release mechanics underneath that regime. [The AI Governance Layer](https://customlabs.io/governance/)
- The Agentic Delivery Playbook covers how a change gets built and merged with a fleet of coding agents. This page starts once that change is ready to ship. [The Agentic Delivery Playbook](https://customlabs.io/agentic-delivery/)
- The Delivery Record publishes six real defects that shipped on this codebase and the build-time check each one left behind. It's the same records-and-traceability argument this page makes, evidenced from our own history instead of asserted. [The Delivery Record](https://customlabs.io/delivery-record/)
- The golden-set gate pattern is the CI mechanics behind the gate surface: what turns "we have an eval set" into "a regression can't merge." [Pattern: Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/)
- Trace-first [observability](https://customlabs.io/glossary/observability/) is the infrastructure every watch control on this page depends on: one trace ID per request, propagated through every hop, so a regression is a specific span instead of a vague complaint. [Pattern: Trace-first observability](https://customlabs.io/patterns/trace-first-observability/)

### Sources

- [Google DORA - DORA Research](https://dora.dev/research/) The research programme behind the delivery metrics our release and delivery pages measure. Retrieved 2026-08-24.
- [Google - Implementing SLOs](https://sre.google/workbook/implementing-slos/) The method for setting a service-level objective our release scoreboard follows. Retrieved 2026-08-24.
- [Google - Error Budget Policy](https://sre.google/workbook/error-budget-policy/) The error-budget policy pattern our release gate borrows. Retrieved 2026-08-24.

Not sure your rollback would actually work

A Ship Audit checks this release path against your specific system and tells you which control is real, versus which one has never been exercised.

[Book a Ship Audit →](https://customlabs.io/diagnostic/ship-audit/) [See the eval stack →](https://customlabs.io/evals/)

Questions

## Before you ship the next change.

What teams ask us before they build a real release process for an AI system.

01 Isn't this just CI/CD with different names? +

Mechanically, much of it looks like ordinary CI/CD: a gate, a ramp, a rollback. The part that's genuinely different is that you can't diff the output of a model change the way you diff a function's return value, so the gate has to run an eval suite instead of a test suite, and reversal has to account for an index and a prompt that drift independently of the code.

02 We just edit the prompt in a dashboard. Do we really need a release process for that? +

If the prompt change can reach real users without going through a gate or a rollback path, then yes, treat it as a release. The blast radius of a bad prompt edit is exactly the same as a bad code deploy; the only thing missing is usually the process, not the risk.

03 What's the difference between this and the Operate stage in the handbook? +

Operate is about detecting a failure once a system is already live: unbounded loops, silent tool failures, a stale index. This page is about the mechanics of shipping the change in the first place, how it gets gated, ramped and reversed, so fewer of those failures reach production ramped to 100% before anyone notices.

04 How do you roll back a retrieval index that already got rebuilt? +

You don't roll back a rebuild in place; you version the index build itself, the same way you'd version a model, and pin the previous build as the fallback. That's the whole argument for shipping prompt, index and tool schemas as one bundle: reverting means switching back to a previous, complete bundle, not trying to undo a rebuild after the fact.

05 What counts as a severity-1 release incident if the system never actually crashes? +

A harmful or materially wrong answer reaching a real user, at scale or in a high-stakes context, counts the same as an outage would. The severity ladder here starts from confidently wrong output rather than from downtime, because that's the failure mode an AI system is actually prone to.

06 Do we need all six surfaces before we ship anything? +

No. A golden-set gate and a documented rollback path cost little and catch most of the risk on a first release. Scale the rest, the canary cohort discipline, the drift thresholds, the incident ladder, to how much exposure and how many independent moving parts (model, prompt, index, tools) the system actually has.

Operate what you just shipped

A rollback that leaves the index untouched is not a real rollback, and an unpulled kill switch is not a safeguard. The Operate stage covers what a release needs after it ships.

[Read the Operate stage →](https://customlabs.io/handbook/operate/) [Talk to us →](https://customlabs.io/contact/)