What breaks in production, and how do we see it before the user does?

What actually happens at this stage

Everything in this stage is a variant of the same problem: production has inputs and conditions the build and evaluate stages didn't see, and the system needs a way to notice when it's operating outside what it was actually tested for, rather than answering confidently anyway. That's not a criticism of the eval suite — no eval set covers every future input — it's the reason observability exists as a distinct discipline from evals rather than a smaller version of the same thing.

Two of the most common operate-stage failures are really the same defect at different layers: an agent that loses track of its own progress, and a tool call that fails silently. An unbounded agent loop keeps retrying a failed approach because the failed attempt sits in context looking exactly like fresh information to reason from — the fix is an explicit step and token budget enforced by the harness, not the model, with a forced terminal state when the budget runs out. Silent tool failure is the same root cause one layer down: a tool that returns HTTP 200 with an error buried in the body gives the model no structural reason to distinguish "nothing matched" from "something broke," so it reports the absence of data as a fact.

Retrieval-specific failures compound because they're invisible until the source and the index diverge: a stale index keeps serving a document that was deleted months ago because nothing ever told the index the source changed, and a chunk boundary that splits a table or procedure in half means the answer that's definitely in the document never makes it into any single retrieved chunk. Both look, from the outside, like "the model got it wrong" — they're actually upstream data problems that no amount of prompt tuning fixes.

Prompt injection via retrieved content deserves specific attention once an agent can both retrieve untrusted text and call real tools, because the model reads one undifferentiated stream of tokens with no privileged channel marking "instruction from us" apart from "text we retrieved." A document in your own knowledge base that reads like a command is functionally indistinguishable to the model from an instruction in the system prompt. There is no complete model-level fix for this today — the actual defense is least-privilege tool scoping and a human checkpoint before anything irreversible, so a successful injection has a small blast radius instead of a catastrophic one.

The operational discipline that ties all of this together is trace-first observability: capturing every prompt, retrieval, and tool call so a regression is a specific span to inspect, not a vague complaint that "the AI got worse." Set up before launch, it's what turns each of the failure modes above from an open-ended incident into a named, detectable, fixable pattern — and it's the same infrastructure the evaluate stage's after-ship half depends on, not a separate system bolted on later.

The common mistake

How do you know this stage is finished?

• Every agent loop has an enforced step/token budget and a forced terminal state, not an open-ended run.
• Tool responses carry an explicit, checkable status the model can branch on — no error-in-a-200-body.
• Retrieval index freshness is tracked as a metric, with a defined tombstone path for deleted source documents.
• Every tool is scoped to least privilege, and irreversible actions have a human checkpoint.
• Trace-first observability is live before launch, not added after the first incident.

What backs this up