What breaks in production, and how do we see it before the user does?
A system that passed every eval still meets a production environment full of stale data, ambiguous tool results, and untrusted retrieved content it was never tested against. Operate is the stage that catches what ships anyway.
What actually happens at this stage#
Everything in this stage is a variant of the same problem: production has inputs and conditions the build and evaluate stages didn't see, and the system needs a way to notice when it's operating outside what it was actually tested for, rather than answering confidently anyway. That's not a criticism of the eval suite. No eval set covers every future input. It's the reason observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. exists as a distinct discipline from evalsAn eval suite is a versioned set of test cases that measures whether outputs are good. rather than a smaller version of the same thing.
Two of the most common operate-stage failures are really the same defect at different layers: an agent that loses track of its own progress, and a tool call that fails silently. An unbounded agent loop keeps retrying a failed approach because the failed attempt sits in context looking exactly like fresh information to reason from. The fix is an explicit step and token budget enforced by the harness, not the model, with a forced terminal state when the budget runs out. Silent tool failure is the same root cause one layer down: a tool that returns HTTP 200 with an error buried in the body gives the model no structural reason to distinguish "nothing matched" from "something broke," so it reports the absence of data as a fact.
Retrieval-specific failures compound because they're invisible until the source and the index diverge: a stale index keeps serving a document that was deleted months ago because nothing ever told the index the source changed, and a chunk boundary that splits a table or procedure in half means the answer that's definitely in the document never makes it into any single retrieved chunk. Both look, from the outside, like "the model got it wrong." They're actually upstream data problems that no amount of prompt tuning fixes.
Prompt injectionPrompt injection is untrusted input crafted to override a model's system prompt or task. via retrieved content deserves specific attention once an agent can both retrieve untrusted text and call real tools, because the model reads one undifferentiated stream of tokens with no privileged channel marking "instruction from us" apart from "text we retrieved." A document in your own knowledge base that reads like a command is functionally indistinguishable to the model from an instruction in the system prompt. There is no complete model-level fix for this today. The actual defense is least-privilege tool scoping and a human checkpoint before anything irreversible, so a successful injection has a small blast radius instead of a catastrophic one.
The operational discipline that ties all of this together is trace-first observability: capturing every prompt, retrieval, and tool call so a regression is a specific span to inspect, not a vague complaint that "the AI got worse." Set up before launch, it's what turns each of the failure modes above from an open-ended incident into a named, detectable, fixable pattern. It's the same infrastructure the evaluate stage's after-ship half depends on, not a separate system bolted on later.
The common mistake#
The most common mistake at this stage is treating a production incident as a one-off prompt fix rather than asking which of these named failure modes it actually was. That means the same defect quietly recurs a few weeks later under a different symptom, because the underlying cause (an unbounded loop, a silent tool failure, a stale index) was never actually addressed.
How do you know this stage is finished?#
- Every agent loop has an enforced step/token budget and a forced terminal state, not an open-ended run.
- Tool responses carry an explicit, checkable status the model can branch on, with no error hidden inside a 200 response body.
- Retrieval index freshness is tracked as a metric, with a defined tombstone path for deleted source documents.
- Every tool is scoped to least privilege, and irreversible actions have a human checkpoint.
- Trace-first observability is live before launch, not added after the first incident.
What backs this up#
Insights
- Why Your AI Agent Works in the Demo and Stalls in Production
The production-stall pattern behind several of this stage's named failure modes.
- Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against
The full threat model behind the injection failure mode below.
- What Actually Moved in the EU AI Act's 2026 Deadline Change
What actually moved in the EU AI Act's 2026 timeline change, and why the register and the traces are worth building on the earlier schedule regardless.
Patterns
- Trace-first observability
The single piece of infrastructure every failure mode above is easier to catch with in place.
Failure modes
- Stale index serves deleted content
The retrieval-freshness failure this stage's ingest checkpoint exists to catch.
- Chunk boundary splits the answer
A retrieval defect that looks like a model mistake from the outside.
- Unbounded agent loop
Why "it ran for 40 minutes and produced nothing" needs a budget, not a better prompt.
- Tool argument hallucination
What a loose tool schema lets a model invent when it can't recall a real value.
- Silent tool failure
The other half of the unbounded-loop problem, one layer down: a tool that fails without saying so.
- Context overflow drops the task
What a naive truncation strategy quietly deletes first: usually the instructions.
- Injection via retrieved content
The specific threat that makes least-privilege tool scoping non-negotiable.
Glossary
- Context Rot
The slow-motion version of this stage's context-overflow failure mode: quality degrading well before the window is technically full.
- Context Isolation
The delegation boundary this stage's tool-scoping discipline depends on once more than one agent or subtask shares a single incident.
- Observability
What turns "the AI got worse" into a specific, inspectable span.
- Shadow AI
What it means when a workaround shows up on this stage's own monitoring, and why it points at a gap in the paved road first.
- Prompt Injection
The attack this stage's tool-scoping discipline exists to bound, not eliminate.
- Confused Deputy
What a gateway holding one shared credential for several downstream tools is exposed to the moment it forwards that credential without checking who it was actually issued for.
- Data Processing Agreement (DPA)
What actually names your model provider as a processor of the personal data this stage's guardrailsGuardrails are the checks that keep an LLM or agent inside acceptable bounds in production. protect.
- Data Residency
Where the data this stage is protecting is actually processed, checked per provider, not assumed from your primary region.
- Human Oversight
The checkpoint design this stage assumes exists: a reviewer who can actually disagree, not a rubber stamp with a name attached.
- Canary Release
The progressive-exposure mechanism this stage's monitoring is what actually makes safe to run.
- Output Drift
The slow-motion version of the regressions this stage's observability is built to catch.
- Model Deprecation
What forces a release even when nobody on the team asked for a change: a provider's own clock.
- System Card
The one document this stage's monitoring and disclosure work should already be keeping current.
More
- AI Security Review
The full checklist InfoSec, privacy, risk, and procurement run against a system operating in production. Prepare it before you book the review, not after a rejection.
- The Delivery Record
What this stage's own discipline actually catches, evidenced from this site's own build history rather than argued in the abstract.
- The AI Governance Layer
The standing regime this stage's records and monitoring discipline feeds into: six surfaces and 24 named controls for the year after the system ships, well past the week it does.
- The AI Release Path
How a change actually ships into this stage's environment: gated, ramped gradually, watched, and reversible, so fewer of the failure modes above reach 100% of traffic before anyone notices.
- The Agent Adoption Playbook
This stage's tool-scoping and monitoring discipline, extended past one system to the whole organisation running it: who owns the paved road, and what unsanctioned use is actually telling you.
- MCP in Production
The operations surface this stage's logging and monitoring discipline owes an MCPMCP is an open standard for connecting LLM applications to tools and data sources. server specifically, including the incident runbook for the case where the broken component is someone else's server.
- Context Engineering
The compaction, memory and integrity surfaces behind this stage's own failure modes: an unbounded loop, a stale index, and injection via retrieved content all trace back to what was and wasn't managed in the window.
Questions on this stage#
What comes up before and during operate.
01 How do we stop an agent that runs forever?
A hard step and token budget enforced by the harness, not the model, with a forced terminal state (success, failure, or escalation) when the budget is hit. The model has no reliable internal sense of "I've tried this six times, stop."
Link to this answer: How do we stop an agent that runs forever?02 Can a document in our own knowledge base actually hijack our agent?
Yes, if the agent can retrieve untrusted content and call real tools. The model can't structurally tell an instruction from retrieved text. Scope every tool to least privilege and require a human checkpoint on anything irreversible; there's no complete model-level fix.
Link to this answer: Can a document in our own knowledge base actually hijack our agent?03 Why does our AI cite a policy we deleted months ago?
Your index was built once at ingest and nothing ever told it the source changed. Move to change-data-capture ingest and add tombstones for deletes so the index doesn't silently drift from the source.
Link to this answer: Why does our AI cite a policy we deleted months ago?A Ship Audit checks your actual system against every stage of this handbook and hands back a written, prioritized plan.