# Shipping to Production Source: https://customlabs.io/topics/shipping-to-production/ Updated: 2026-09-14 Topics # Shipping to Production The gap between a working notebook and a system that runs reliably and cheaply at scale is usually a bigger lift than the original prototype. This is our work on closing that gap. Start with The AI Release Path if you own shipping AI to production. It's for engineers who need a reliable way to ship a change and reverse it fast. In the Handbook [01 Decide](https://customlabs.io/handbook/decide/)[02 Design](https://customlabs.io/handbook/design/)[03 Build](https://customlabs.io/handbook/build/)[04 Evaluate](https://customlabs.io/handbook/evaluate/)[05 Operate](https://customlabs.io/handbook/operate/)[06 Cost](https://customlabs.io/handbook/cost/) ## Guides ### The Agentic Delivery Playbook The operating model for running delivery with a fleet of coding agents: six control surfaces and 22 practices, measured by the metrics that catch review debt early. 26 min read [Read →](https://customlabs.io/agentic-delivery/) ### The Agent Tool Interface Agents fail at the tool interface more often than at the model. The design rules for building an interface that holds up in production. 26 min read [Read →](https://customlabs.io/tool-design/) ### The AI Release Path How to ship a change to an AI system and ramp it safely. Reverse it fast if it turns out to be wrong. 34 min read [Read →](https://customlabs.io/release/) ### The Agent Adoption Playbook The pilot worked. How the rest of your engineers actually start using it, and the numbers that separate real adoption from seat activation. 33 min read [Read →](https://customlabs.io/adoption/) ### MCP in Production [MCP](https://customlabs.io/glossary/model-context-protocol/) standardizes the wire format. Identity, the catalog and the trust boundary are still yours to build, plus a ten-question gate for a new server. 42 min read [Read →](https://customlabs.io/mcp/) ## Architectures ### Agentic workflow runner for a back-office queue An agent that works a queue of cases end to end, inside a bounded loop with a human checkpoint on anything it cannot cleanly undo. 7 min read [Read →](https://customlabs.io/architectures/agentic-workflow-runner/) ### Document extraction with a human review loop Turn inbound documents into structured records, with a person reviewing only the cases the model is actually unsure about. 6 min read [Read →](https://customlabs.io/architectures/document-extraction-pipeline/) ### An assistant inside a product you already ship An assistant embedded in a product you already ship, scoped to that signed-in user's own data and permissions. 7 min read [Read →](https://customlabs.io/architectures/in-product-copilot/) ### A coding-agent delivery fleet A control plane for running many coding agents at once without them colliding on the same files or losing track of what any of it costs. 7 min read [Read →](https://customlabs.io/architectures/agent-delivery-fleet/) ## Comparisons July 22, 2026 ### Agents vs Pipelines: When Autonomy Is Worth the Reliability Cost Agentic autonomy and deterministic pipelines aren't a spectrum of ambition. They trade flexibility against reliability, and here's which your workflow needs. 7 min read [Read →](https://customlabs.io/compare/agents-vs-pipelines/) ## Insights July 24, 2026 ### Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against [Prompt injection](https://customlabs.io/glossary/prompt-injection/) can't be filtered away: the model can't reliably tell instructions from data. Here's the actual threat model and the controls that hold up. 10 min read [Read →](https://customlabs.io/insights/prompt-injection-threat-model/) July 19, 2026 ### Why Your AI Agent Works in the Demo and Stalls in Production An agent that nails the demo stalls in production because reliability compounds across steps. Here's the math, the real failure modes, and how to ship anyway. 8 min read [Read →](https://customlabs.io/insights/ai-agents-stall-in-production/) July 11, 2026 ### What AI Actually Costs in Production, by Workload A reproducible benchmark of cost per successful outcome across four common AI workloads, with every token assumption, price, and overhead multiplier shown. 9 min read [Read →](https://customlabs.io/insights/ai-cost-benchmark-by-workload/) July 7, 2026 ### Your RAG Demo Lied to You Retrieval that looks flawless on ten clean PDFs falls apart on a real corpus. Here's why, and what evaluating retrieval quality actually requires. 7 min read [Read →](https://customlabs.io/insights/your-rag-demo-lied/) June 30, 2026 ### What an AI Feature Actually Costs in Production Token costs that look trivial in a demo compound fast at scale. Here's how to make cost a first-class metric instead of a surprise on the invoice. 8 min read [Read →](https://customlabs.io/insights/what-ai-actually-costs/) April 14, 2026 ### From Notebook to Production: What Actually Breaks The gap between a working AI demo and a production feature is auth, latency, cost and fallbacks. Here's how we close it without a rewrite. 7 min read [Read →](https://customlabs.io/insights/notebook-to-production/) ## Case studies June 30, 2026 ### From Notebook to Production: An Extraction Model You Could Trust A healthtech's extraction model dazzled in a notebook, then hallucinated on real traffic. An eval harness cut [hallucinations](https://customlabs.io/glossary/hallucination/) from 12% to 3% in 8 weeks. 7 min read [Read →](https://customlabs.io/case-studies/extraction-notebook-to-production/) ## Patterns ### Intent router to specialists A cheap, fast classifier reads each request and routes it to one narrow specialist agent, instead of one god-agent that carries every tool. Each specialist holds only the tools, context, and instructions its job needs. The router's only job is picking the right specialist, and the specialist's only job is the task it was built for. [Read →](https://customlabs.io/patterns/intent-router-to-specialists/) ### Bounded agent loop An agent loop runs under an explicit budget: a maximum step count, a token ceiling, and a wall-clock limit. A termination contract forces every run to end in one of a few named states: success, failure, or escalation. When the budget runs out before the model reaches one of those states on its own, the harness ends the run itself, instead of letting it continue indefinitely. [Read →](https://customlabs.io/patterns/bounded-agent-loop/) ### Typed tool contract Every tool argument is defined by a strict JSON schema: enums for known value sets, validated patterns for IDs, required fields where genuinely needed. There is no free-text catch-all surface. A call that fails validation is rejected with a structured, actionable error the model can act on, never silently coerced or passed through to execution. [Read →](https://customlabs.io/patterns/typed-tool-contract/) ### Human checkpoint before irreversible actions Every tool the agent can call is scoped to the narrowest permission the task genuinely needs. Any action that can't be cleanly undone, such as a refund, a delete, or an external message, requires an explicit human confirmation before it executes, beyond a plausible-looking model decision. The two controls compound: least privilege bounds the blast radius of anything that slips through, and the checkpoint stops the worst actions from executing at all. [Read →](https://customlabs.io/patterns/human-checkpoint-before-irreversible/) ## Failure modes ### Stale index serves deleted content Your retrieval index was built once at ingest and never told the source changed. When a document is edited or deleted, nothing re-embeds the new version or tombstones the old chunk. The stale vector keeps scoring well and keeps getting served with confidence, giving the reader no signal that it is out of date. [Read →](https://customlabs.io/failure-modes/stale-index-serves-deleted-content/) ### Unbounded agent loop The agent has no step budget, no token budget, and no way to recognize it is stuck. A failing tool call stays in its context and keeps looking like a reasonable next thing to try, so it keeps trying variations of the same failed approach until something external (a timeout, a bill, a human) stops it. [Read →](https://customlabs.io/failure-modes/unbounded-agent-loop/) ### Tool argument hallucination Loose tool schemas (free-form string IDs, everything optional) give the model room to fill a gap with something plausible-looking instead of something real. With no server-side validation catching the mismatch before execution, a confidently invented ID reaches a system that expects a real one. [Read →](https://customlabs.io/failure-modes/tool-argument-hallucination/) ### Silent tool failure The tool returned HTTP 200 with an error message in the body, or an empty result set, and the agent read the absence of data as evidence rather than as a failure. Nothing in the response forced a distinction between "nothing matched" and "something broke." [Read →](https://customlabs.io/failure-modes/silent-tool-failure/) ### Context overflow drops the task As the conversation grows, a naive truncation strategy drops the oldest messages to stay under the context window. The oldest messages are exactly where the system prompt and the original task state usually live, so the agent keeps running with no memory of what it was actually supposed to do. [Read →](https://customlabs.io/failure-modes/context-overflow-drops-the-task/) ### Injection via retrieved content Yes. Retrieved content arrives on the same channel as instructions, so a document, ticket, or webpage crafted (or compromised) to contain commands can have the model execute them with its real tool permissions. The system has no built-in way to tell 'instruction from us' apart from 'text we retrieved.' [Read →](https://customlabs.io/failure-modes/injection-via-retrieved-content/) ## Tools ### AI Readiness Scorecard A fast, honest read on whether your data, infra, and process are ready to ship AI. [Read →](https://customlabs.io/tools/ai-readiness/)