From Notebook to Production: What Actually Breaks

A working AI notebook demo proves the model can do the task; it says nothing about surviving real users, real traffic, and real failure. Five things break on the way to production: auth and multi-tenant data isolation, latency under concurrent load, cost at scale, provider fallbacks and degraded modes, and evals instead of vibes. None of it requires a rewrite. Treat the demo as the spec and build the production plumbing around it deliberately.

Every AI project we’ve been brought into looks the same at the start: someone on the team built a genuinely impressive demo in a notebook. The model does the thing. The prompt is tuned. Everyone in the room is convinced. Then someone asks “when can this ship?” and the honest answer is: not yet, because none of the hard parts have been touched.

What actually breaks between a notebook and production?

Auth and multi-tenancy. A notebook has one user: you. Production has many, each with different permissions, different data they’re allowed to see, and different rate limits you owe them. Retrieval-augmented features are the sharpest version of this problem: if your vector index isn’t partitioned by tenant, you will eventually leak one customer’s documents into another customer’s answer. It’s the default outcome of copying a notebook’s retrieval loop straight into a multi-tenant app.

Latency turns into a problem the moment there’s load on the system. A single call to a frontier model in a notebook takes a few seconds and nobody minds. At production traffic, that same call, multiplied across concurrent users, queues up, and a UI with no story for “the model is still thinking” reads as broken. Streaming responses, optimistic UI, and background job patterns for anything slower than a couple of seconds are what separate a feature people use from one they abandon after the first slow response.

Cost, at scale. Token costs that look trivial in a demo (a few cents per call) get uncomfortable fast once a feature is live for thousands of users doing it repeatedly. The fix is knowing which parts of the pipeline actually need the expensive model and which can run on something smaller, cached, or skipped entirely, rather than treating “use a cheaper model” as a blanket policy. That requires instrumentation you don’t get from a notebook: per-call cost tracking, broken down by feature and by user, from day one.

Fallbacks and degraded modes matter just as much. Every model provider has outages, rate limits, and the occasional bad day where quality quietly drops. A notebook has no opinion about what happens when the API call fails; production absolutely must, because “the feature is just broken” is not an acceptable failure mode for something customers depend on. That means provider fallback chains, sensible retry/backoff, and, critically, a designed-in degraded experience rather than a stack trace.

Evals, not vibes. In the notebook, “does this work” is judged by eyeballing a handful of outputs. That’s fine for a proof of concept and actively dangerous for anything you intend to keep changing. Once a feature ships, you will tweak the prompt, swap a model, or adjust a retrieval parameter. Without a standing eval suite, you have no way to know if that change made things better or quietly worse for a case you didn’t happen to glance at.

What does a real path to production look like?

None of the above requires a rewrite from scratch. It requires treating the demo as a spec, not a deliverable, and building the boring infrastructure around it deliberately:

• Auth and tenant isolation wired through the same access-control layer as the rest of your app, rather than a parallel system for “the AI part.”
• Observability on every model call: latency, token cost, and outcome, tagged by feature and tenant, so you can answer “is this working” with data instead of impressions.
• A fallback chain across providers (or at minimum across model tiers) with retry and backoff tuned to your actual traffic patterns.
• An eval suite that runs in CI on every change to a prompt, model, or retrieval parameter, giving a concrete, repeatable answer to “did this get better or worse.”
• A rollout plan that ships to a slice of traffic first (often 5–10%), so the first time a genuinely adversarial input hits the feature, it isn’t hitting all your users at once.

This is also, not coincidentally, the same discipline that makes it possible to run many AI agents against a codebase at once without them colliding. It’s the exact problem we built CodeHerder to solve for coding agents specifically: real-time visibility into what every agent is doing, and coordination so they don’t clobber each other’s work. The pattern generalizes. Once you’re running AI processes at production scale, whether that’s agents writing code or a chat feature answering customer questions, you need to actually see what’s happening, not just trust that it’s fine.

How does this fit into a client engagement?

When we take on an AI integration, we treat the notebook as the requirements document, not the codebase. The build starts from the production constraints (auth, latency budget, cost ceiling, fallback behavior, eval criteria) and works backward to the model call, rather than starting from the model call and hoping the rest holds up. It’s slower to get a first demo this way, often by a few weeks. It’s considerably faster to get to something you can actually leave running.

If you’ve got a demo that’s stalled at “impressive but not shipped,” that’s usually a sign the hard part hasn’t been scoped yet — not that the idea is wrong. Happy to look at where the gap actually is.