Why Your AI Agent Works in the Demo and Stalls in Production

An agent that nails a scripted demo stalls in production because reliability compounds multiplicatively across steps, not additively. A 95%-per-step agent chaining 20 tool calls succeeds end-to-end only about 36% of the time. The demo ran one happy path; production runs the long tail. Shorten the loop, make each step independently verifiable, and build step-level evals before adding more autonomy.

Every agent demo we’ve watched follows the same arc: five or six steps, a clean input, a live audience, and a confident finish. Nobody in the room asks what happens on step 14 when a tool call returns malformed JSON, because step 14 never comes up. It comes up in production, on day three, for a customer nobody was watching over the shoulder of. By then the agent has already taken three more steps on top of the bad one.

Why does a 95%-reliable agent fail more than half the time?

A single 95%-reliable step succeeds 95% of the time, by definition. Chain N of them, and if each step’s success is independent, the end-to-end success rate is 0.95 raised to the N — a modeled, illustrative curve, not a measured production number, but the compounding is real:

Steps chained	Modeled success rate
5	≈ 77%
10	≈ 60%
20	≈ 36%

This is the single fact that most agent demos hide by construction. A demo is one run through a script the builder already knows works, so it exercises exactly one path through the distribution, the happy one. Production doesn’t get to pick its path; it runs every input a real user sends, across every step in the chain, and the 20th step inherits the accumulated failure probability of the 19 before it. An agent that “basically works” at the demo’s 5-step depth can be a coin flip by the time a real workflow stretches it to 15 or 20.

What actually breaks between the demo and production?

Tool-call reliability. Every external call an agent makes (an API, a database query, another model) is another factor in the product above, and each one fails in its own way: malformed arguments the agent generates with high confidence, a vendor schema that drifted since the prompt was written, a flaky or slow endpoint that times out mid-chain. A demo hits two or three tools, always in a state the builder tested; production hits however many the workflow needs, in whatever state the upstream system happens to be in that day.

Non-determinism. The same input can produce a different plan on a different run, because the model is sampling, not executing a fixed program. That shows up as retry loops that burn budget without converging, or as a plan that quietly branches away from the intended workflow into something that looks plausible step-by-step and ends up nowhere useful.

State and context erosion. Long transcripts accumulate the agent’s own prior outputs as context, and an early mistake doesn’t stay contained. It becomes input to every subsequent step, so the agent reasons forward from a premise that was already wrong. By step 15, an agent can be confidently, coherently building on an error it made at step 3, with nothing in the transcript flagging that the foundation is bad.

How do you make an agent reliable enough to ship?

The most reliable fix for a known workflow is often less agent. A fixed pipeline lets the model handle only the steps that genuinely need judgment, with deterministic code handling everything else, the same “skip the build if you don’t need it” honesty we apply to build-vs-buy decisions generally. Free-form autonomy is worth the added reliability cost only where the workflow is genuinely unpredictable, not the default choice.

An eval suite that only grades the final output can’t tell you which of 20 steps is dragging the product down. Step-level evals independently verify each step’s output against a known-good expectation, so a regression at step 6 gets caught at step 6 instead of surfacing three weeks later as a dip in end-to-end success.

Guardrails matter too: constrained tool schemas that reject malformed arguments before they execute, output validation on every step rather than trusting the model’s own confidence, bounded retries so a stuck step fails loudly instead of looping silently, and human checkpoints before any irreversible action like a payment, a delete, or an email sent to a customer.

Step-level tracing closes the loop. If a failure can’t be traced to the specific step and tool call that caused it, every fix is a guess. Observability on each step’s input, output, latency, and outcome turns “the agent did something wrong somewhere” into a specific, fixable defect.

What this doesn’t tell you (honest limits)

The 0.95^n model assumes every step is independent and uniformly reliable, and neither assumption fully holds in a real system. Failures are often correlated: a bad upstream API response can degrade several downstream steps at once rather than failing them independently. A step with a retry or a self-correction pass also has a higher effective reliability than a single blind attempt, since the agent gets a second look before the failure propagates. The precise percentages here are illustrative, not a production SLA. The point of the curve is its shape: reliability decays multiplicatively, so step count is a lever you can pull, not a fixed cost of doing agentic work.

Shipping one anyway

None of this is an argument against agents. It’s an argument against shipping one the way it looked in the demo. The agents we’ve put into production run on the same discipline as any other reliability-critical system: fewer steps than the demo used, an eval at each one, and tracing that shows exactly where a chain broke instead of just that it did. That’s also the exact problem CodeHerder is built to make visible for coding agents specifically: real-time tracing of what every agent is doing, step by step, so a stall in production is a specific, findable defect instead of a mystery. If you’ve got an agent that impressed in the room and stalled on real traffic, that’s usually a step-count and observability problem, not a model problem. Happy to help find which step it is.

For a diagnostic breakdown of the specific failures behind a stalled agent, see the field guide entries on unbounded agent loops, tool argument hallucination, silent tool failure, and context overflow dropping the task.