What breaks in production, and how do we see it before the user does?
A system that passed every eval still meets a production environment it never saw. That environment holds stale data, ambiguous tool results and untrusted retrieved content. Operate is the stage that catches what ships anyway.
What happens at this stage#
Everything in this stage is a variant of one problem. Production has inputs and conditions the build and evaluate stages did not see. The system needs a way to notice when it operates outside what it was tested for. Answering confidently anyway is the failure to avoid. That's not a criticism of the eval suite. No eval set covers every future input. It's the reason observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. exists as a distinct discipline from evalsAn eval suite is a versioned set of test cases that measures whether outputs are good. rather than a smaller version of the same thing.
Two of the most common operate-stage failures are one defect at different layers. One is an agent that loses track of its own progress. The other is a tool call that fails silently. An unbounded agent loop keeps retrying a failed approach because the failed attempt sits in context looking exactly like fresh information to reason from. The fix is an explicit step and token budget that the harness enforces. The model does not enforce it. A forced terminal state applies when the budget runs out. Silent tool failure has the same root cause one layer down. A tool returns HTTP 200 with an error buried in the body. The model then has no structural reason to tell "nothing matched" from "something broke". It reports the absence of data as a fact.
Retrieval-specific failures compound because they stay invisible until the source and the index diverge. A stale index keeps serving a document that was deleted months ago, because nothing told the index the source changed. A chunk boundary can split a table or procedure in half. The answer that is in the document then never reaches any single retrieved chunk. Both look, from the outside, like "the model got it wrong." They're actually upstream data problems that no amount of prompt tuning fixes.
Prompt injectionPrompt injection is untrusted input crafted to override a model's system prompt or task. via retrieved content deserves attention once an agent can both retrieve untrusted text and call real tools. The model reads one undifferentiated stream of tokens. No privileged channel separates "instruction from us" from "text we retrieved". A document in your own knowledge base that reads like a command looks the same to the model as an instruction in the system prompt. There is no complete model-level fix for this today. The defense is least-privilege tool scoping plus a human checkpoint before anything irreversible. A successful injection then has a small blast radius.
Trace-first observability ties all of this together. It captures every prompt, retrieval and tool call. A regression then becomes a specific span to inspect, and the complaint that "the AI got worse" goes away. Set it up before launch. It turns each failure mode above from an open-ended incident into a named, detectable, fixable pattern. It's the same infrastructure the evaluate stage's after-ship half depends on, not a separate system bolted on later.
The common mistake#
The most common mistake at this stage is treating a production incident as a one-off prompt fix. The better question is which of these named failure modes it was. The same defect then recurs a few weeks later under a different symptom. Nobody addressed the underlying cause, such as an unbounded loop, a silent tool failure or a stale index.
How do you know this stage is finished?#
- Every agent loop has an enforced step/token budget and a forced terminal state, not an open-ended run.
- Tool responses carry an explicit, checkable status the model can branch on, with no error hidden inside a 200 response body.
- Retrieval index freshness is tracked as a metric, with a defined tombstone path for deleted source documents.
- Every tool is scoped to least privilege, and irreversible actions have a human checkpoint.
- Trace-first observability is live before launch, not added after the first incident.
What backs this up#
Insights
- Why Your AI Agent Works in the Demo and Stalls in Production
The production-stall pattern behind several of this stage's named failure modes.
- Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against
The full threat model behind the injection failure mode below.
- What Moved in the EU AI Act's 2026 Deadline Change
What moved in the EU AI Act's 2026 timeline change. It also explains why the register and the traces are worth building on the earlier schedule.
Patterns
- Trace-first observability
The single piece of infrastructure every failure mode above is easier to catch with in place.
Failure modes
- Stale index serves deleted content
The retrieval-freshness failure this stage's ingest checkpoint exists to catch.
- Chunk boundary splits the answer
A retrieval defect that looks like a model mistake from the outside.
- Unbounded agent loop
Why "it ran for 40 minutes and produced nothing" needs a budget, not a better prompt.
- Tool argument hallucination
What a loose tool schema lets a model invent when it can't recall a real value.
- Silent tool failure
The other half of the unbounded-loop problem, one layer down: a tool that fails without saying so.
- Context overflow drops the task
What a naive truncation strategy quietly deletes first: usually the instructions.
- Injection via retrieved content
The specific threat that makes least-privilege tool scoping non-negotiable.
Glossary
- Context Rot
The slow-motion version of this stage's context-overflow failure mode: quality degrading well before the window is technically full.
- Context Isolation
The delegation boundary this stage's tool-scoping discipline depends on once more than one agent or subtask shares a single incident.
- Observability
What turns "the AI got worse" into a specific, inspectable span.
- Shadow AI
What it means when a workaround shows up on this stage's own monitoring. It also explains why that points at a gap in the paved road first.
- Prompt Injection
The attack this stage's tool-scoping discipline exists to bound, not eliminate.
- Confused Deputy
What a gateway holding one shared credential for several downstream tools is exposed to. The risk starts when it forwards that credential without checking who it was issued for.
- Data Processing Agreement (DPA)
What actually names your model provider as a processor of the personal data this stage's guardrailsGuardrails are the checks that keep an LLM or agent inside acceptable bounds in production. protect.
- Data Residency
Where the data this stage is protecting is actually processed, checked per provider, not assumed from your primary region.
- Human Oversight
The checkpoint design this stage assumes exists: a reviewer who can actually disagree, not a rubber stamp with a name attached.
- Canary Release
The progressive-exposure mechanism this stage's monitoring is what actually makes safe to run.
- Output Drift
The slow-motion version of the regressions this stage's observability is built to catch.
- Model Deprecation
What forces a release even when nobody on the team asked for a change: a provider's own clock.
- System Card
The one document this stage's monitoring and disclosure work should already be keeping current.
More
- AI Security Review
The full checklist InfoSec, privacy, risk, and procurement run against a system operating in production. Prepare it before you book the review, not after a rejection.
- The Delivery Record
What this stage's own discipline actually catches, evidenced from this site's own build history rather than argued in the abstract.
- The AI Governance Layer
The standing regime that this stage's records and monitoring discipline feed into. It has six surfaces and 24 named controls for the year after the system ships and well past launch week.
- The AI Release Path
How a change ships into this stage's environment. It is gated, ramped gradually, watched and reversible. Fewer of the failure modes above then reach 100% of traffic before anyone notices.
- The Agent Adoption Playbook
This stage's tool-scoping and monitoring discipline, extended past one system to the whole organisation running it. It asks who owns the paved road and what unsanctioned use tells you.
- MCP in Production
The operations surface that this stage's logging and monitoring discipline owes an MCPMCP is an open standard for connecting LLM applications to tools and data sources. server. It includes the incident runbook for when the broken component is someone else's server.
- Context Engineering
The compaction, memory and integrity surfaces behind this stage's failure modes. An unbounded loop, a stale index and injection via retrieved content all trace back to the window. Each one comes from something the team did not manage.
Questions on this stage#
What comes up before and during operate.
01 How do we stop an agent that runs forever?
A hard step and token budget that the harness enforces. The model does not. A forced terminal state (success, failure or escalation) applies when the budget is hit. The model has no reliable internal sense of "I've tried this six times, stop."
Link to this answer: How do we stop an agent that runs forever?02 Can a document in our own knowledge base actually hijack our agent?
Yes, if the agent can retrieve untrusted content and call real tools. The model can't structurally tell an instruction from retrieved text. Scope every tool to least privilege and require a human checkpoint on anything irreversible; there's no complete model-level fix.
Link to this answer: Can a document in our own knowledge base actually hijack our agent?03 Why does our AI cite a policy we deleted months ago?
Your index was built once at ingest and nothing ever told it the source changed. Move to change-data-capture ingest and add tombstones for deletes so the index doesn't silently drift from the source.
Link to this answer: Why does our AI cite a policy we deleted months ago?A Ship Audit checks your actual system against every stage of this handbook and hands back a written, prioritized plan.