From Notebook to Production: What Breaks

A working AI notebook demo proves the model can do the task; it says nothing about surviving real users, real traffic, and real failure. Five things break on the way to production. Auth and multi-tenant data isolation, latency under concurrent load, cost at scale, provider fallbacks and degraded modes, and evals instead of vibes. None of it requires a rewrite. Treat the demo as the spec and build the production plumbing around it deliberately.

Every AI project we’ve been brought into looks the same at the start: someone on the team built a impressive demo in a notebook. The model does the thing. The prompt is tuned. Everyone in the room is convinced. Then someone asks “when can this ship?” and the honest answer is: not yet, because none of the hard parts have been touched.

What breaks between a notebook and production?

Auth and multi-tenancy. A notebook has one user: you. Production has many, each with different permissions, different data they’re allowed to see, and different rate limits you owe them. Retrieval-augmented features are the sharpest version of this problem. If your vector index isn’t partitioned by tenant, you will eventually leak one customer’s documents into another customer’s answer. It’s the default outcome of copying a notebook’s retrieval loop straight into a multi-tenant app.

Latency turns into a problem the moment there’s load on the system. A single call to a frontier model in a notebook takes a few seconds and nobody minds. At production traffic, that same call gets multiplied across concurrent users and queues up. A UI with no story for “the model is still thinking” reads as broken. Streaming responses, optimistic UI, and background job patterns matter for anything slower than a couple of seconds. Those patterns separate a feature people use from one they abandon after the first slow response.

Cost, at scale. Token costs that look trivial in a demo (a few cents per call) get uncomfortable fast. That happens once a feature is live for thousands of users doing it repeatedly. The fix is knowing which parts of the pipeline need the expensive model. Other parts can run on something smaller, cached, or skipped entirely. Don’t treat “use a cheaper model” as a blanket policy for every part. That requires instrumentation you don’t get from a notebook: per-call cost tracking, broken down by feature and by user, from day one.

Fallbacks and degraded modes matter just as much. Every model provider has outages, rate limits, and the occasional bad day where quality drops. A notebook has no opinion about what happens when the API call fails. Production must have one, because “the feature is just broken” is not an acceptable failure mode for something customers depend on. That means provider fallback chains, sensible retry/backoff, and, critically, a designed-in degraded experience rather than a stack trace.

Evals, not vibes. In the notebook, “does this work” is judged by eyeballing a handful of outputs. That’s fine for a proof of concept and actively dangerous for anything you intend to keep changing. Once a feature ships, you will tweak the prompt, swap a model, or adjust a retrieval parameter. Without a standing eval suite, you have no way to know if a change helped. You also won’t know if it broke a case you didn’t happen to glance at.

What does a real path to production look like?

None of the above requires a rewrite from scratch. It requires treating the demo as a spec, not a deliverable, and building the boring infrastructure around it deliberately:

• Auth and tenant isolation wired through the same access-control layer as the rest of your app, rather than a parallel system for “the AI part.”
• ObservabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. on every model call: latency, token cost, and outcome, tagged by feature and tenant. You can then answer “is this working” with data instead of guesswork.
• A fallback chain across providers (or at minimum across model tiers) with retry and backoff tuned to your actual traffic patterns.
• An eval suite runs in CI on every change to a prompt, model, or retrieval parameter. It gives a concrete, repeatable answer to “did this get better or worse.”
• A rollout plan ships to a slice of traffic first, often 5–10%. That way, the first adversarial input hits only part of your users, not all of them at once.

This is also, not coincidentally, the same discipline that makes it possible to run many AI agents against a codebase at once without them colliding. It’s the exact problem we built CodeHerder to solve for coding agents specifically. It gives real-time visibility into what every agent is doing, and coordination so they don’t clobber each other’s work. The pattern generalizes. Once you’re running AI processes at production scale, whether that’s agents writing code or a chat feature answering questions, you need to see what’s happening. You need to know, not just trust, that it’s fine.

How does this fit into a client engagement?

When we take on an AI integration, we treat the notebook as the requirements document, not the codebase. The build starts from the production constraints (auth, latency budget, cost ceiling, fallback behavior, eval criteria) and works backward to the model call. That’s the opposite of starting from the model call and hoping the rest holds up. It’s slower to get a first demo this way, often by a few weeks. It’s considerably faster to get to something you can leave running.

If your demo is stalled at “impressive but not shipped,” that usually means the hard part hasn’t been scoped yet. It doesn’t mean the idea is wrong. Happy to look at where the gap is.