CustomLabs
Agentic delivery

How we ship with a fleet of coding agents: a first-hand account.

This page covers six control surfaces and 22 practices. It also covers five ways a fleet fails and the metrics that catch review debt early. This operating model sits behind every merge request on this site. It is not a sales pitch.

Updated First published

26 min read

Markdown

Most writing about managing AI coding agents is vendor marketing for an orchestration product. This page is different. It describes the operating model this studio runs. The same model landed the merge request that shipped this page. Every fleet we have run, and every fleet we have read about, shows one finding. Once generation is cheap, review throughput is the bottleneck. Model capability is not.

Each of the six control surfaces has a named owner and prevents a specific failure. If you skip one, the fleet does not get slower. It gets faster at producing work that nobody can trust.

The operating model

Six control surfaces.#

Each one governs a different thing. A fleet with a missing surface fails without noise. The failure looks like progress.

Diagram in six lanes, one per control surface: task definition; isolation; stage gates; review throughput; durable memory; and spend and blast radius. Each lane holds one node that names the thing that surface governs. No edges connect the surfaces; each one covers separate ground. TASK DEFINITION ISOLATION STAGE GATES REVIEW THROUGHPUT DURABLE MEMORY SPEND & BLAST RADIUS A checkable 'done' Shared mutable state Named exit artifact Review queue growth Re-litigated decisions Ceiling on spend
Six control surfaces run at once. Each one covers a different way a fleet of coding agents can fail.
01

Task definition

Owner: The person who writes the task: a tech lead or the engineer who requests the work

The acceptance criteria a task carries before an agent starts, and whether 'done' has a checkable definition

  • The agent asks clarifying questions mid-run or guesses, because the task never said what 'done' means
  • Work that satisfies the prompt but misses the goal ships anyway
  • Two agents get the same vague ask and read it two ways. Both results get merged
02

Isolation

Owner: The owner of the sandbox and orchestration layer: the fleet operator

Whether two agents can touch the same mutable state at the same time

  • Two agents edit the same file in parallel and one overwrites the other's work
  • A blanket `git add -A` on one agent's branch picks up another agent's uncommitted work
  • Two unrelated tasks run against one shared dev database and leave it half-migrated
03

Stage gates

Owner: The orchestration harness and the pipeline definition. No single agent owns them

The sequence a task moves through, and what a fresh context inherits at each hand-off

  • An agent 'finishes' a task in its own reasoning but never opens a request, so the work never lands
  • A stage that cannot advance sits idle and burns wall-clock time. It should release the slot
  • A planning mistake survives every later stage, because review inherited the same context and the same blind spot
04

Review throughput

Owner: The owner of the review queue: a tech lead or the review-stage configuration

How many tasks can be in flight before the review queue grows without bound

  • Merge lag climbs every week while 'tasks shipped' climbs too: throughput up, delivery down
  • PRs get rubber-stamped, because there are more of them than one person can read
  • The same category of bug ships again and again, because review looks at the diff and never at the pattern behind it
05

Durable memory

Owner: The curator of workspace memory: often a lead engineer or the fleet operator

Whether every later session has to re-argue a decision that was made once

  • The same architectural debate happens again, nearly word for word, a few tasks later
  • A convention agreed six tasks ago drifts, because nobody saved the reasoning behind it anywhere durable
  • Institutional knowledge lives in one tool's per-session context and disappears when that session ends
06

Spend & blast radius

Owner: The owner of the harness's budget defaults: the infra or platform owner

The ceiling on any single loop's steps, tokens, wall-clock time, and cost, and the gate every landing path must pass

  • A retry storm turns a two-dollar task into a two-hundred-dollar task before anyone notices
  • An agent pushes directly to the default branch and CI finds the break after it is live
  • A runaway loop continues because nothing watched the clock. Only an exit condition watched it, and that condition never fires
Practice bank

22 practices, six surfaces.#

Filter by maturity (crawl, walk, run). Then copy the visible list as a Markdown adoption checklist.

01 Task definition

Task specificity caps output quality. Model capability does not. Give an agent "add authentication" and it will produce something that runs. That something may have little to do with the need. The agent filled the gaps itself, as anyone would when the ask is underspecified. A task is a spec. It has background and a named goal. It also has explicit constraints and an acceptance criterion. Someone can check it without asking the requester what they meant. A chat-prompt task is faster to write. It also gives you a confident, well-executed answer to the wrong question.

Why
It prevents confident, excellent work against a goal that nobody agreed to. Without it, 'looks right' is the only bar anyone can check.
Signal you don't have it
Reviewers argue whether a PR 'does what was asked' after the code is written. They should argue before anyone starts.
The control
A required acceptance-criteria field on every task, checked by a verification command that the task names.
Evidence
The task template with a mandatory, non-empty acceptance-criteria field. Add a sample of tasks where the criterion is a command.
Why
In a single paragraph, 'constraint' and 'nice-to-have' blur together. The agent cannot tell what it must leave unchanged.
Signal you don't have it
An agent's hand-off comment restates the task almost word for word, because it had to guess which sentence was the real constraint.
The control
A task schema with named, separately required fields in place of one free-text box.
Evidence
The task-creation template, and a sample task where scope, goal, and constraints are distinct fields.
Why
The task is the interface between what you meant and what gets built. A chat prompt replaces that interface and gives it nothing to work from.
Signal you don't have it
Every completed task needs a follow-up round of "that's not quite what I meant" before it is usable.
The control
A gate before the code stage that bounces underspecified tasks back for a written spec. The agent does not start guessing.
Evidence
The rejected-task log, or a sample of tasks bounced at intake for missing required fields.
Why
Whichever agent reaches the verify stage decides an unnamed verification step. The step can change from task to task.
Signal you don't have it
Two agents on similar tasks report 'done' with two different definitions of passing.
The control
A required verification-command field. The verify stage runs it the same way, whichever agent wrote the code.
Evidence
The verify-stage config or script that runs the task's own named command. A generic smoke test does not qualify.

02 Isolation

Fleets of coding agents fail on shared mutable state long before they fail on reasoning. Two agents edit the same file. Two tasks change a shared dev database at once. One agent's blanket add picks up another agent's uncommitted work. The model did not cause any of these. Better prompting cannot fix them, because the failure is architectural: two writers share one piece of state with no ownership boundary. The fix is plain and mandatory. Give each task one branch, one working tree, and one sandbox. A task that misbehaves then has a blast radius of one task.

Why
Fleets fail on shared mutable state. They rarely fail on reasoning. A shared checkout is the most common way two agents collide.
Signal you don't have it
A commit shows up on your branch that you did not write and cannot explain.
The control
One sandbox provisioned per task at task start and torn down at task end. Concurrent tasks share no filesystem.
Evidence
The sandbox-provisioning config showing a strict one-to-one task-to-sandbox mapping.
Why
A broad add picks up whatever another agent left uncommitted in a shared or adjacent workspace. It credits their work to your task or loses it.
Signal you don't have it
A PR contains changes to files that the task description never mentioned.
The control
Commit hygiene enforced by convention and spot-checked at review. The agent stages named files explicitly and never uses a wildcard add.
Evidence
A sample of recent commits showing explicit file lists in place of blanket adds.
Why
A second, unrelated task can leave a shared dev database half-migrated. The result looks like a bug in your own change until someone loses an afternoon to it.
Signal you don't have it
A task fails intermittently, and the fix that 'worked' was to rerun it. Nobody changed any code.
The control
Per-task ephemeral environments, or a locking and queueing layer in front of any environment that cannot be duplicated.
Evidence
The environment-provisioning setup showing task-scoped database instances with no sharing.
Why
A later stage that can inspect an earlier stage's open session recreates the shared-mutable-state problem that isolation exists to prevent. The problem only moves up one level.
Signal you don't have it
To debug a review-stage failure, you must re-open the original code-stage session. You cannot just read its hand-off comment.
The control
Stage boundaries enforced at the infrastructure level: a new context and a new process, with a read of the written artifact only.
Evidence
The stage-transition mechanism showing that each stage spawns a fresh session with no reference to the prior session's live state.

03 Stage gates

A fleet needs named stages with named exit artifacts. The agent does not decide for itself when it is done. Plan produces a plan. Code produces a diff and an open request. Review produces a verdict. Merge produces a merged change. Verify produces a pass or a routed-back failure. Every stage starts a fresh context. That context inherits only what the previous stage wrote down. A stage that cannot advance releases the slot. It does not loop in place and wait for input that will not come. Skip a stage, or let an agent self-certify that a later stage is unneeded, and the fleet reverts. It becomes one long, ungated context that nobody can audit.

Why
It prevents the agent that 'finished' the work in its own reasoning but produced nothing that the next stage or a human can act on.
Signal you don't have it
A task shows recent activity in its logs, but nothing else exists anywhere: no request, no comment, no artifact.
The control
A stage gate that transitions only when its named artifact is present and checked. The agent claiming completion is not enough.
Evidence
The stage-transition rule that enforces artifact presence before advancing, and its rejection log.
Why
If review inherits the context of planning, a planning mistake survives every later stage. Review shares the blind spot that produced it.
Signal you don't have it
Review consistently agrees with the code stage's own self-assessment and catches nothing that code missed.
The control
Each stage runs in a new session. The only input from the prior stage is its written comment. The transcript stays behind.
Evidence
The session-boundary implementation, and a sample hand-off comment showing what crosses the boundary.
Why
An idle stage that holds a slot blocks capacity that another task could use. From outside, it looks like progress until someone checks.
Signal you don't have it
A task's last-touch timestamp is old, its status has not changed. Nothing is queued behind it.
The control
An explicit release action that frees the execution slot without advancing the task. The action is logged with a reason.
Evidence
The release log showing reasons, and the resulting slot reallocation.
Why
The stage that an agent skips is the one that would have caught what it missed. Self-assessment is a poor substitute for an independent stage.
Signal you don't have it
A change reaches merge with no review-stage artifact anywhere in its history.
The control
The pipeline enforces stage order structurally. A task cannot enter a later stage until the prior stage's exit artifact is recorded.
Evidence
The pipeline definition itself, and an audit of a sample of merged tasks showing every stage represented.

04 Review throughput

Once generation is cheap, review is the bottleneck. More agents without more review capacity do not raise throughput. They grow a queue of unread work. The fix has layers. Machine review runs first. It is adversarial and independent, and it catches what a first pass misses before a human opens the diff. A human then reviews decisions and the diffs that carry risk. Nobody has the attention to read every line of every PR. A fleet sized past its review capacity ships no faster. It converts engineering hours into a growing pile of merge requests that nobody has read.

Why
Generation scales far faster than review capacity. Uncapped concurrency moves the bottleneck into a growing queue that has a faster-looking front end.
Signal you don't have it
Merge lead time trends upward for weeks while 'tasks completed' trends upward too.
The control
A concurrency ceiling tied to a measured reviewer capacity. Revisit it when that capacity changes. A number picked once will go stale.
Evidence
The concurrency-limit config, and the review-lead-time trend it is set against.
Why
A human cannot read every line of every PR past a handful of agents. A machine pass that tries to find problems catches most of what a human would otherwise find by hand. A rubber-stamp pass does not.
Signal you don't have it
Every PR reaches human review in about the same state it left the code stage. Nothing is caught or fixed in between.
The control
A dedicated review stage staffed by an independent reviewer, agent or process, whose job is to find problems. It runs before the human gate.
Evidence
The review-stage findings log showing issues caught and fixed before human review.
Why
A reviewer cannot see their own blind spot. A second independent pass with a different vantage point is the only reliable way to catch it.
Signal you don't have it
The same category of bug recurs across several merged changes, and every one of them passed review.
The control
A second review pass with a distinct lens, such as security, correctness, or scope. It is required before merge on changes above a defined risk threshold.
Evidence
The multi-pass review configuration, and a sample showing disagreement caught between passes.
Why
If you spread finite human attention evenly across all output, the highest-risk change gets the same scrutiny as the lowest. That is too little for the first and wasted on the second.
Signal you don't have it
A reviewer's calendar is full of low-risk approvals while a risky change waits in the same queue.
The control
Risk-tiering that routes low-risk, well-covered changes to a lighter gate. It reserves full human attention for the changes that need it.
Evidence
The risk-tiering rule, and the review-time distribution it produces across tiers.

05 Durable memory

A fleet that forgets re-argues the same decision every session, at the same cost. It reaches a different answer half the time. The reasoning that settled the question last time lived in a context windowThe context window is the maximum text, measured in tokens, a model can consider at once. that no longer exists. Conventions drift the same way. Every fresh agent invents its own idiom for a problem the last agent already solved, because nothing durable told it otherwise. The fix is memory that outlives any single session. It is shared and versioned at workspace level. People curate it so that it does not grow into an unread pile. A per-tool cache dies when the process exits and does not meet this need.

Why
Code shows what was decided. It does not show why. The next agent that meets an adjacent decision re-argues it from scratch without the reasoning that settled it the first time.
Signal you don't have it
The same architectural debate happens again, nearly word for word, a few tasks later.
The control
A memory-write step that is part of finishing a task. It is not an optional extra for whoever remembers.
Evidence
The memory store showing decisions with their stated reasoning.
Why
A convention that lives in one session context is invisible to every other agent. It is gone when that session ends.
Signal you don't have it
Two agents working in the same week produce two different idioms for the same problem. Neither knows the other exists.
The control
A memory layer that every agent in the fleet can address, whichever tool or session wrote to it.
Evidence
The shared memory store's access pattern showing reads from sessions that did not write the original entry.
Why
Nobody trusts memory that nobody curates, so nobody reads it. That is the same as having none.
Signal you don't have it
New entries keep getting written that contradict older entries nobody removed.
The control
A review step, human or agent, that reconciles memory on a schedule in the same way that code gets reviewed.
Evidence
A changelog of memory edits: merges, corrections, and retirements.

06 Spend & blast radius

Every agent loop needs an explicit ceiling on steps, tokens, wall-clock time, and cost. The harness enforces the ceiling. The model cannot, because it has no reliable internal sense of when to stop after six failed tries. Every landing path also needs a gate that no agent can bypass. Never push directly to the default branch. CI is the arbiter. A budget that runs out forces a terminal state. The loop does not idle forever. Skip either control and the failure takes the same shape twice. A retry storm turns a two-dollar task into a two-hundred-dollar task. Or an unreviewed 3am change goes unnoticed until it is live.

Why
A model has no reliable internal sense of when to stop after six failed tries. An unbounded loop keeps retrying a failed approach, because the failed attempt sits in context and looks like fresh information.
Signal you don't have it
A single task's cost or run time is ten times that of a similar task, with no matching increase in complexity.
The control
A budget enforced by the orchestration harness, independent of the model, with a forced terminal state when the budget is hit.
Evidence
The budget configuration, and the distribution of terminal states (success, failure, escalated) across recent runs.
Why
A direct push skips every gate the fleet relies on (review, tests, verification) in one step. One push can put an unreviewed change into production.
Signal you don't have it
A commit on the default branch has no corresponding merge request anywhere in the history.
The control
Branch protection that rejects direct pushes at the git-host level.
Evidence
The branch-protection rule configuration, and a git history showing one merge request per landed change.
Why
A retry-amplified task that costs forty times its estimate stays hidden in a monthly aggregate until the invoice arrives. Per-task tracking catches it while it is still one task and has not become a pattern across many.
Signal you don't have it
The first sign of a cost problem is a monthly bill. No per-task alert fired.
The control
A per-task cost ledger checked against budget in real time, with an alert or hard stop when a task exceeds it.
Evidence
The cost-ledger dashboard or report, broken out per task.
Five ways a fleet fails

These are operating-model failures.#

Each of these looks like an agent capability problem from the outside. None of them is one.

Review debt

Looks like: Merge lag climbs while "throughput" improves. The fleet completes more tasks, and a longer queue of them waits to ship.

Guardrail: Cap in-flight tasks at the review capacity you have. Run machine review before human review.

Green tests, broken feature

Looks like: The agent writes the tests that it then passes. An all-green suite says nothing about whether the feature is right.

Guardrail: Have a different context author the acceptance criteria before the code stage. Make verification its own stage.

Convention drift

Looks like: Every agent invents its own idiom for the same problem, because nothing durable told it what the last agent decided.

Guardrail: Durable memory plus house conventions that the fleet reads. Lint and CI enforce them. Hope does not.

Silent scope creep

Looks like: The agent improves things that nobody asked for. The diff is twice the size that the task needed.

Guardrail: Name the scope in the task. Review bounces diffs that touch files outside the stated files.

Retry-amplified spend

Looks like: A loop costs forty times the estimate. The team finds out on the monthly invoice and not while it happens.

Guardrail: Hard ceilings, a per-task budget and "release, don't stall" in place of idle looping.

The metrics that matter

Six numbers and how each one misleads.#

You can measure each of these today. Each one can look better than it is.

Human-to-agent ratio

Tells you
How many concurrent agents one reviewer can hold before quality drops.
Lies when
A high ratio looks efficient until merge lag and rework rate show that the reviewer is the bottleneck.
Act by
Add review capacity before you add another agent. Capacity is a person or a machine-review pass.

Review lead time

Tells you
How long finished work waits, ready to merge, before it ships.
Lies when
A short median hides the tasks that are stuck in the queue behind it. The average does not count them.
Act by
Watch the trend on the slow tail. A growing p90 means the queue is backing up while the median still looks fine.

First-pass merge rate

Tells you
The percentage of tasks that land with no rework round. It shows how often the first attempt is good enough.
Lies when
A rising rate can mean better task definition. It can also mean reviewers wave weaker work through to keep the number up.
Act by
Pair it with escaped-defect rate before you trust it. If both rise together, the reviewer is giving up and the fleet is not improving.

Round-trips per task

Tells you
How many times a task bounces between stages before it lands: the rework rate.
Lies when
Zero round-trips is not always good. The review pass may have been too weak to send anything back.
Act by
Investigate each task with three or more round-trips on its own. The cause is almost always task definition. Agent capability is rarely the cause.

Cost per merged change

Tells you
What a shipped change costs, with every attempt and every round of rework included.
Lies when
Cost per API call, or per task started, undercounts it. Retries and abandoned attempts vanish from that number.
Act by
Divide total spend by merged changes. Do not divide by tasks started. Watch the trend before the monthly invoice does.

Escaped-defect rate

Tells you
Post-merge fixes attributable to agent work. It shows whether the review stage caught what it should have caught.
Lies when
Zero escaped defects early on usually means too little traffic has hit the change. It does not prove that review is airtight.
Act by
Track it per stage-gate configuration change. A rate that climbs after you loosen a review gate is the signal to tighten the gate again.
What we run

Claims you can verify.#

This page has no invented metrics. It has no throughput multipliers and no client names. It states what this repository's own history proves.

Sources

  1. Google DORA - DORA Capabilities Catalog

    The capability list this research associates with high-performing delivery teams. Retrieved 2026-08-24.

  2. Google DORA - DORA Research

    The research programme behind the delivery metrics our release and delivery pages measure. Retrieved 2026-08-24.

  3. Google - Implementing SLOs

    The method for setting a service-level objective our release scoreboard follows. Retrieved 2026-08-24.

Run this yourself

CodeHerder is the task registry and per-task sandbox layer that runs this operating model day to day. It is the how behind the why on this page.

Questions

Before you run this on Monday.#

What teams ask us before adopting this operating model.

01 Is this just vibe coding at scale?

No. Vibe coding is a chat prompt and a hope. This model uses a written spec with a machine-checkable acceptance criterion, an isolated sandbox, and a named stage sequence. A verification stage runs after review. A human does not eyeball the output and call it good.

Link to this answer: Is this just vibe coding at scale?
02 What's the bottleneck?

Review, once generation is cheap. More agents do not raise throughput if review capacity stays fixed. They grow an unread queue with a faster-looking front end. Machine review before human review is the fix. See the review-throughput surface above.

Link to this answer: What's the bottleneck?
03 Do we need a framework or an orchestrator?

Something has to implement the six surfaces. Isolation, stage gates, and spend ceilings are hard to run by convention alone. The framework is an implementation detail. Judge any tool by whether it gives you these six things. Ignore its feature list.

Link to this answer: Do we need a framework or an orchestrator?
04 How do you stop two agents breaking each other's work?

Isolation: one task, one branch, one working tree, one sandbox. It is the least glamorous of the six surfaces. It causes the most damage when you skip it. See the isolation surface above.

Link to this answer: How do you stop two agents breaking each other's work?
05 Does this hold up on regulated or audited code?

This operating model covers how work gets delivered. It does not cover compliance sign-off. Those are a different set of questions, and they are real. Our AI Security Review has the InfoSec, privacy, risk, and procurement checklist that a regulated system must also clear.

Link to this answer: Does this hold up on regulated or audited code?
06 Does it replace engineers?

No. Review throughput is the human bottleneck by design. Do not engineer it away. The review and stage-gate surfaces exist to apply a senior engineer's judgment efficiently across more work.

Link to this answer: Does it replace engineers?
07 How do we start on Monday?

Pick two surfaces and leave the other four. Start with task definition (require an acceptance criterion on every task) and spend and blast radius (a hard ceiling and branch protection). Both are crawl-maturity and cheap. Together they prevent the two failure modes that do the most damage.

Link to this answer: How do we start on Monday?
Run the fleet on purpose

Fleet control needs more than agents that pass their own tests, since a green suite can still hide a broken feature. CodeHerder runs all six surfaces this playbook maps, across the whole fleet.

Source: https://customlabs.io/agentic-delivery/

↑↓ navigate ↵ select esc close