What shape is the system: pipeline, agent, retrieval, or none of the above?

What actually happens at this stage

Design is where most of the long-term reliability of an AI system gets decided, and it happens before any model is called in anger. The central fork is agent versus pipeline: a pipeline is a fixed sequence of steps where the model fills in one or two of them, predictable and easy to test because the control flow doesn't change; an agent is a loop where the model decides what happens next, powerful for open-ended tasks and much harder to bound, test, and debug. Most systems that get called "an AI agent" in a pitch deck would be more reliable, cheaper, and easier to eval as a pipeline — reach for a real agent only when the task's steps genuinely can't be known in advance.

If any part of the system needs to answer from your own data, retrieval enters the design, and it deserves the same deliberateness as the agent-vs-pipeline call. The default architecture is chunk, embed, retrieve by similarity, rerank, generate — and every one of those five verbs is a place a demo-passing system quietly breaks in production. Chunking that ignores document structure splits tables and procedures in half; embeddings encode topical resemblance far better than exact identifiers, dates, or negation; a system that looks solid on ten demo queries can fail badly on the identifier-shaped queries real users actually ask.

The vector-database-versus-pgvector fork belongs here too, and it's smaller than it looks from the outside: if you already run Postgres and your corpus is in the low millions of vectors, pgvector removes an entire piece of infrastructure to operate for a real but bounded performance cost; a dedicated vector database earns its keep once scale, filtering complexity, or query latency requirements outgrow what an extension bolted onto your primary database can deliver.

Whichever shape you land on, design it to be model-agnostic from the start, not as a retrofit after the first provider price change or deprecation notice. That means an abstraction layer between your application and any single model provider — not a big one, but a real one — so a workload can route to whichever model is cheapest or most capable for that specific call, and a provider outage or pricing change becomes a config change rather than a rewrite. Almost every team that skips this does so because it looks like premature abstraction on day one; almost every team that skips it retrofits it within a year, under worse conditions than they'd have chosen for themselves.

Whatever shape emerges — agent, pipeline, retrieval, or a combination — should come out of this stage as a small number of named, deliberate patterns, not an emergent pile of prompts. An intent router in front of specialist agents instead of one god-agent; a bounded loop with an explicit step and token budget instead of an open-ended one; typed tool contracts instead of free-form string arguments. We hit exactly this coordination problem operating fleets of coding agents ourselves — it's the reason CodeHerder exists, as a shared task registry and per-task cost ledger instead of a wall of uncoordinated terminal tabs. None of these are exotic; they're the standard shapes for the failure modes this stage is trying to design around before they happen, and adopting the two or three your system actually needs beats adopting all of them.

The common mistake

How do you know this stage is finished?

• You've explicitly chosen agent, pipeline, retrieval, or a combination — and can say why the alternative shapes were rejected.
• If retrieval is involved, chunking respects document structure and you've checked identifier/negation queries specifically, not just topical ones.
• The vector-store decision (pgvector vs. a dedicated database) is sized to your actual scale, not defaulted to whichever is trendiest.
• There's a model-agnostic layer between your application and any single provider, even a thin one.
• The design maps to two or three named patterns your system actually needs — not every pattern in the library.

What backs this up