# 11 patterns for AI systems that survive production. Source: https://customlabs.io/patterns/ Updated: 2026-09-07 Patterns # 11 patterns for AI systems that survive production. The prescriptive companion to the failure-mode field guide. That's what breaks; this is how to build so it doesn't. Named, opinionated, with the honest trade-offs and the signals to instrument. Failure mode → pattern that prevents it If you have this failure mode Build with this pattern [Stale index serves deleted content](https://customlabs.io/failure-modes/stale-index-serves-deleted-content/) [Change-data-capture ingest](https://customlabs.io/patterns/change-data-capture-ingest/) [Chunk boundary splits the answer](https://customlabs.io/failure-modes/chunk-boundary-splits-the-answer/) [Structure-aware chunking](https://customlabs.io/patterns/structure-aware-chunking/) [Similarity is not relevance](https://customlabs.io/failure-modes/similarity-is-not-relevance/) [Retrieve-then-rerank](https://customlabs.io/patterns/retrieve-then-rerank/) [Unbounded agent loop](https://customlabs.io/failure-modes/unbounded-agent-loop/) [Bounded agent loop](https://customlabs.io/patterns/bounded-agent-loop/) [Tool argument hallucination](https://customlabs.io/failure-modes/tool-argument-hallucination/) [Typed tool contract](https://customlabs.io/patterns/typed-tool-contract/) [Silent tool failure](https://customlabs.io/failure-modes/silent-tool-failure/) [Typed tool contract](https://customlabs.io/patterns/typed-tool-contract/) , [Trace-first observability](https://customlabs.io/patterns/trace-first-observability/) [Context overflow drops the task](https://customlabs.io/failure-modes/context-overflow-drops-the-task/) [Intent router to specialists](https://customlabs.io/patterns/intent-router-to-specialists/) [Vibes-based prompt regression](https://customlabs.io/failure-modes/vibes-based-prompt-regression/) [Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/) [Judge prefers its own output](https://customlabs.io/failure-modes/judge-prefers-its-own-output/) [Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/) [Retry-amplified spend](https://customlabs.io/failure-modes/retry-amplified-spend/) [Bounded agent loop](https://customlabs.io/patterns/bounded-agent-loop/) , [Model cascade](https://customlabs.io/patterns/model-cascade/) [Prompt cache never hits](https://customlabs.io/failure-modes/prompt-cache-never-hits/) [Stable-prefix prompt caching](https://customlabs.io/patterns/stable-prefix-prompt-caching/) [Injection via retrieved content](https://customlabs.io/failure-modes/injection-via-retrieved-content/) [Human checkpoint before irreversible actions](https://customlabs.io/patterns/human-checkpoint-before-irreversible/) ## Agent control flow Agent control flow · 6 min read ### Intent router to specialists A cheap, fast classifier reads each request and routes it to one narrow specialist agent, instead of one god-agent that carries every tool. Each specialist holds only the tools, context, and instructions its job needs. The router's only job is picking the right specialist, and the specialist's only job is the task it was built for. [Read the pattern →](https://customlabs.io/patterns/intent-router-to-specialists/) Agent control flow · 6 min read ### Bounded agent loop An agent loop runs under an explicit budget: a maximum step count, a token ceiling, and a wall-clock limit. A termination contract forces every run to end in one of a few named states: success, failure, or escalation. When the budget runs out before the model reaches one of those states on its own, the harness ends the run itself, instead of letting it continue indefinitely. [Read the pattern →](https://customlabs.io/patterns/bounded-agent-loop/) ## Retrieval Retrieval · 5 min read ### Structure-aware chunking Chunk boundaries follow the document's own structure: headings, table rows, list items, section boundaries, instead of a fixed token count. Each chunk carries its parent heading or identifying context in its own body. A table row is never separated from its header, and a step is never separated from the procedure it belongs to. The chunk that gets embedded is always a complete unit of meaning, not an arbitrary slice. [Read the pattern →](https://customlabs.io/patterns/structure-aware-chunking/) Retrieval · 5 min read ### Retrieve-then-rerank A cheap, high-recall first pass pulls a wide candidate set of 50 to 100 documents likely to contain the right answer. It's usually vector search, optionally fused with keyword search. A precision reranker, usually a cross-encoder that scores the query and each candidate jointly, then re-sorts that set. It applies a relevance floor, allowed to return nothing rather than force a weak match to the top. [Read the pattern →](https://customlabs.io/patterns/retrieve-then-rerank/) Retrieval · 5 min read ### Change-data-capture ingest The ingest pipeline subscribes to the actual change events of the source system, such as a webhook, a CMS publish hook, or a database trigger. That replaces re-crawling on a fixed schedule. An edit triggers a re-embed within minutes. A delete writes a tombstone that excludes the old chunks from retrieval immediately, before a full re-index even runs. [Read the pattern →](https://customlabs.io/patterns/change-data-capture-ingest/) ## Reliability & guardrails Reliability & guardrails · 5 min read ### Typed tool contract Every tool argument is defined by a strict JSON schema: enums for known value sets, validated patterns for IDs, required fields where genuinely needed. There is no free-text catch-all surface. A call that fails validation is rejected with a structured, actionable error the model can act on, never silently coerced or passed through to execution. [Read the pattern →](https://customlabs.io/patterns/typed-tool-contract/) Reliability & guardrails · 6 min read ### Human checkpoint before irreversible actions Every tool the agent can call is scoped to the narrowest permission the task genuinely needs. Any action that can't be cleanly undone, such as a refund, a delete, or an external message, requires an explicit human confirmation before it executes, beyond a plausible-looking model decision. The two controls compound: least privilege bounds the blast radius of anything that slips through, and the checkpoint stops the worst actions from executing at all. [Read the pattern →](https://customlabs.io/patterns/human-checkpoint-before-irreversible/) ## Evals & observability Evals & observability · 5 min read ### Golden-set gate in CI A fixed, human-labelled set of real cases runs automatically in CI on every prompt or model change. Each case carries a specific, checkable expected property, not a vibe. A drop against the current pass-rate threshold blocks the merge the same way a failing unit test would. A regression gets caught before release, not after a user complaint. [Read the pattern →](https://customlabs.io/patterns/golden-set-gate-in-ci/) Evals & observability · 5 min read ### Trace-first observability One trace ID follows a single request across every hop it takes: retrieval, every model call, every tool call. It's logged with enough detail to reconstruct exactly what happened after the fact. A sampled slice of traces, weighted toward low-confidence or error-flagged runs, lands in a queue a human reviews on a regular schedule. That happens well before any complaint forces someone to go looking. [Read the pattern →](https://customlabs.io/patterns/trace-first-observability/) ## Cost & routing Cost & routing · 5 min read ### Model cascade A cheap, fast model attempts every request first, and a validation signal decides whether that attempt is good enough to return. That signal can be a schema check, a self-reported confidence score, or a downstream eval. Only the fraction of requests the cheap model couldn't handle escalates to a stronger, pricier model. Most traffic never reaches the expensive tier at all. [Read the pattern →](https://customlabs.io/patterns/model-cascade/) Cost & routing · 5 min read ### Stable-prefix prompt caching The prompt is ordered with everything invariant across calls first: system instructions, tool definitions, few-shot examples. Everything that changes per request, such as retrieved chunks, user input, or timestamps, goes last. Prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front is enough to bust the cache for the entire call, so ordering invariant-first is what makes the discount actually apply. [Read the pattern →](https://customlabs.io/patterns/stable-prefix-prompt-caching/) No patterns match that filter yet. Questions ## Which of these do you actually need? The questions that come up before a team decides which patterns to adopt and which to skip. 01 How is this different from the failure-mode field guide? + The failure-mode field guide is diagnostic. It starts from a symptom you already have and works backward to a root cause and a fix. This library is prescriptive: it starts from an architectural decision you make before anything breaks. The cross-reference table above links them directly. Every failure mode traces to at least one pattern that prevents it from happening. 02 Do I need all of these? + No. Most systems need three or four of these, not eleven. Every entry has an explicit "don't use it when" section for exactly this reason. Adopting a pattern your system doesn't need is its own kind of complexity debt. Start from whichever failure mode you're most exposed to, and work from there. 03 Are these specific to one model provider? + No. Every pattern here is a structural or architectural decision. That means a routing layer, a schema, a caching order, a CI gate. Not a prompt tuned to one model's quirks. They hold up whether the model behind them is a frontier API or a self-hosted open-weight model. See our take on staying model-agnostic by design. Next step A Ship Audit tells you which of these patterns your system is actually missing, prioritized against what's most likely to break first. [Book a Ship Audit →](https://customlabs.io/diagnostic/ship-audit/)