# The Applied AI Handbook. Source: https://customlabs.io/handbook/ Updated: 2026-09-05 Handbook # The Applied AI Handbook. Six stages, one lifecycle: the decisions a real AI project actually meets, in the order it meets them, with the studio's full body of work filed under each one. Updated September 5, 2026 · First published July 29, 2026 · [Start with Decide →](https://customlabs.io/handbook/decide/) This is everything we know about shipping applied AI, arranged the way you'll actually meet it: as six decisions, in the order a real project makes them, rather than a list of insights, case studies, and reference pages sorted by publish date. Those decisions are whether to build at all, what shape to build, how to get it to production, how to know it works, what breaks once it's live, and what it actually costs to keep running. Each stage carries its own argument: the decision, the sequence, the mistake most teams make there. It then cites the specific insights, patterns, failure modes, and glossary entries that back it up. Read it front to back if you're scoping a first AI system, or jump straight to whichever stage matches where your project actually is. The handbook runs as six decisions in a fixed order, from whether to build at all through to what it costs to keep running. 01 Decide ## Should we build this with AI at all: build, buy, or skip? Every AI initiative starts with a decision most teams never make explicitly: whether the problem is actually an AI problem. Get this call right and the rest of the build gets easier; get it wrong by default and no amount of good engineering downstream fixes it. 16 sources · 7 min read [Read stage →](https://customlabs.io/handbook/decide/) 02 Design ## What shape is the system: pipeline, agent, retrieval, or none of the above? Once "build" wins, the next decision is architectural: a fixed pipeline, an autonomous agent, a retrieval layer, or some combination. Each shape carries a different reliability profile before you write the first prompt. 27 sources · 8 min read [Read stage →](https://customlabs.io/handbook/design/) 03 Build ## How do we get from a working notebook to a deployable service? A working prototype and a production service are different projects with different scopes. The gap between them is usually a bigger lift than the prototype itself, and it's the stage most timelines quietly underestimate. 16 sources · 6 min read [Read stage →](https://customlabs.io/handbook/build/) 04 Evaluate ## How do we know it works, and how do we keep knowing after every change? Shipping without an [eval suite](https://customlabs.io/glossary/eval-suite/) means every prompt or model change afterward is a guess about whether quality went up or down. Evaluate is the stage that turns that guess into a number. 16 sources · 6 min read [Read stage →](https://customlabs.io/handbook/evaluate/) 05 Operate ## What breaks in production, and how do we see it before the user does? A system that passed every eval still meets a production environment full of stale data, ambiguous tool results, and untrusted retrieved content it was never tested against. Operate is the stage that catches what ships anyway. 31 sources · 8 min read [Read stage →](https://customlabs.io/handbook/operate/) 06 Cost ## What will this actually cost to run, and where does the spend hide? The per-token sticker price on a pricing page is not what an AI feature costs to run. Retries, fallbacks, context growth, and agent loops routinely push real production spend to 5-20x the naive estimate. 11 sources · 6 min read [Read stage →](https://customlabs.io/handbook/cost/) Full contents ### 01 Decide - [Build, Buy, or Skip: A Framework for AI Decisions](https://customlabs.io/insights/build-buy-or-skip/) Insight - [What an AI Feature Actually Costs in Production](https://customlabs.io/insights/what-ai-actually-costs/) Insight - [RAG vs Fine-Tuning: Which One Actually Solves Your Problem](https://customlabs.io/compare/rag-vs-fine-tuning/) Comparison - [Open-Weight Models vs Frontier APIs: The Real Cost/Control Tradeoff](https://customlabs.io/compare/open-weight-vs-frontier-api/) Comparison - [The AI-Native Target That Wasn't: A Two-Week Diligence](https://customlabs.io/case-studies/ai-diligence-flagged-replatform-cost/) Case study - [AI Readiness Scorecard](https://customlabs.io/tools/ai-readiness/) Tool - [FreeTier](https://customlabs.io/products/freetier/) Page - [Choosing an AI Delivery Partner](https://customlabs.io/choosing-a-partner/) Page - [The AI Cost Model](https://customlabs.io/cost/) Page - [The AI Governance Layer](https://customlabs.io/governance/) Page - [Retrieval-Augmented Generation (RAG)](https://customlabs.io/glossary/retrieval-augmented-generation/) Glossary - [Fine-Tuning (vs RAG)](https://customlabs.io/glossary/fine-tuning/) Glossary - [Context Window](https://customlabs.io/glossary/context-window/) Glossary - [Token](https://customlabs.io/glossary/token/) Glossary - [AI Register](https://customlabs.io/glossary/ai-register/) Glossary - [High-Risk AI System](https://customlabs.io/glossary/high-risk-ai-system/) Glossary ### 02 Design - [Agents vs Pipelines: When Autonomy Is Worth the Reliability Cost](https://customlabs.io/compare/agents-vs-pipelines/) Comparison - [Vector Database vs pgvector: Do You Actually Need a Dedicated Store](https://customlabs.io/compare/vector-database-vs-pgvector/) Comparison - [Model-Agnostic by Design](https://customlabs.io/insights/model-agnostic-by-design/) Insight - [Retrieval Pipeline That Actually Cut Support Load](https://customlabs.io/case-studies/retrieval-pipeline-cut-support-load/) Case study - [Intent router to specialists](https://customlabs.io/patterns/intent-router-to-specialists/) Pattern - [Bounded agent loop](https://customlabs.io/patterns/bounded-agent-loop/) Pattern - [Typed tool contract](https://customlabs.io/patterns/typed-tool-contract/) Pattern - [Structure-aware chunking](https://customlabs.io/patterns/structure-aware-chunking/) Pattern - [Retrieve-then-rerank](https://customlabs.io/patterns/retrieve-then-rerank/) Pattern - [CodeHerder](https://customlabs.io/products/codeherder/) Page - [The Agent Tool Interface](https://customlabs.io/tool-design/) Page - [Reference Architectures](https://customlabs.io/architectures/) Page - [Architecture Picker](https://customlabs.io/tools/architecture-picker/) Tool - [Grounded answering over your own documents](https://customlabs.io/architectures/grounded-answering/) Page - [Agentic workflow runner for a back-office queue](https://customlabs.io/architectures/agentic-workflow-runner/) Page - [An assistant inside a product you already ship](https://customlabs.io/architectures/in-product-copilot/) Page - [Model-agnostic inference gateway](https://customlabs.io/architectures/model-gateway/) Page - [Agent (Agentic AI)](https://customlabs.io/glossary/agent/) Glossary - [Tool Calling](https://customlabs.io/glossary/tool-calling/) Glossary - [Model Context Protocol (MCP)](https://customlabs.io/glossary/model-context-protocol/) Glossary - [MCP Gateway](https://customlabs.io/glossary/mcp-gateway/) Glossary - [Tool Contract](https://customlabs.io/glossary/tool-contract/) Glossary - [Context Engineering](https://customlabs.io/glossary/context-engineering/) Glossary - [Chunking](https://customlabs.io/glossary/chunking/) Glossary - [Embeddings](https://customlabs.io/glossary/embeddings/) Glossary - [Vector Search (Semantic Search)](https://customlabs.io/glossary/vector-search/) Glossary - [Model-Agnostic Architecture](https://customlabs.io/glossary/model-agnostic-architecture/) Glossary ### 03 Build - [From Notebook to Production: What Actually Breaks](https://customlabs.io/insights/notebook-to-production/) Insight - [From Notebook to Production: An Extraction Model You Could Trust](https://customlabs.io/case-studies/extraction-notebook-to-production/) Case study - [Change-data-capture ingest](https://customlabs.io/patterns/change-data-capture-ingest/) Pattern - [Human checkpoint before irreversible actions](https://customlabs.io/patterns/human-checkpoint-before-irreversible/) Pattern - [Notebook-to-Production](https://customlabs.io/glossary/notebook-to-production/) Glossary - [Structured Output](https://customlabs.io/glossary/structured-output/) Glossary - [Guardrails](https://customlabs.io/glossary/guardrails/) Glossary - [The Agentic Delivery Playbook](https://customlabs.io/agentic-delivery/) Page - [Document extraction with a human review loop](https://customlabs.io/architectures/document-extraction-pipeline/) Page - [A coding-agent delivery fleet](https://customlabs.io/architectures/agent-delivery-fleet/) Page - [The Agent Adoption Playbook](https://customlabs.io/adoption/) Page - [Golden Path](https://customlabs.io/glossary/golden-path/) Glossary - [MCP in Production](https://customlabs.io/mcp/) Page - [Tool Poisoning](https://customlabs.io/glossary/tool-poisoning/) Glossary - [Context Engineering](https://customlabs.io/context-engineering/) Page - [Compaction](https://customlabs.io/glossary/compaction/) Glossary ### 04 Evaluate - [Evals Before You Ship: Why AI Features Need Tests Too](https://customlabs.io/insights/evals-before-you-ship/) Insight - [Your RAG Demo Lied to You](https://customlabs.io/insights/your-rag-demo-lied/) Insight - [Golden-set gate in CI](https://customlabs.io/patterns/golden-set-gate-in-ci/) Pattern - [Trace-first observability](https://customlabs.io/patterns/trace-first-observability/) Pattern - [Vibes-based prompt regression](https://customlabs.io/failure-modes/vibes-based-prompt-regression/) Failure mode - [Judge prefers its own output](https://customlabs.io/failure-modes/judge-prefers-its-own-output/) Failure mode - [Similarity is not relevance](https://customlabs.io/failure-modes/similarity-is-not-relevance/) Failure mode - [Eval Suite (Evals)](https://customlabs.io/glossary/eval-suite/) Glossary - [LLM-as-Judge](https://customlabs.io/glossary/llm-as-judge/) Glossary - [Hallucination](https://customlabs.io/glossary/hallucination/) Glossary - [Red Teaming](https://customlabs.io/glossary/red-teaming/) Glossary - [Shadow Deployment](https://customlabs.io/glossary/shadow-deployment/) Glossary - [Acceptance Rate](https://customlabs.io/glossary/acceptance-rate/) Glossary - [Quality is a number, not a vibe.](https://customlabs.io/capabilities/#evals) Page - [The Eval Stack](https://customlabs.io/evals/) Page - [The AI Release Path](https://customlabs.io/release/) Page ### 05 Operate - [Why Your AI Agent Works in the Demo and Stalls in Production](https://customlabs.io/insights/ai-agents-stall-in-production/) Insight - [Prompt Injection Is a Data Problem: A Threat Model You Can Ship Against](https://customlabs.io/insights/prompt-injection-threat-model/) Insight - [What Actually Moved in the EU AI Act's 2026 Deadline Change](https://customlabs.io/insights/ai-act-high-risk-deadline-moved/) Insight - [Stale index serves deleted content](https://customlabs.io/failure-modes/stale-index-serves-deleted-content/) Failure mode - [Chunk boundary splits the answer](https://customlabs.io/failure-modes/chunk-boundary-splits-the-answer/) Failure mode - [Unbounded agent loop](https://customlabs.io/failure-modes/unbounded-agent-loop/) Failure mode - [Tool argument hallucination](https://customlabs.io/failure-modes/tool-argument-hallucination/) Failure mode - [Silent tool failure](https://customlabs.io/failure-modes/silent-tool-failure/) Failure mode - [Context overflow drops the task](https://customlabs.io/failure-modes/context-overflow-drops-the-task/) Failure mode - [Injection via retrieved content](https://customlabs.io/failure-modes/injection-via-retrieved-content/) Failure mode - [Trace-first observability](https://customlabs.io/patterns/trace-first-observability/) Pattern - [AI Security Review](https://customlabs.io/security-review/) Page - [The Delivery Record](https://customlabs.io/delivery-record/) Page - [The AI Governance Layer](https://customlabs.io/governance/) Page - [The AI Release Path](https://customlabs.io/release/) Page - [The Agent Adoption Playbook](https://customlabs.io/adoption/) Page - [MCP in Production](https://customlabs.io/mcp/) Page - [Context Engineering](https://customlabs.io/context-engineering/) Page - [Context Rot](https://customlabs.io/glossary/context-rot/) Glossary - [Context Isolation](https://customlabs.io/glossary/context-isolation/) Glossary - [Observability](https://customlabs.io/glossary/observability/) Glossary - [Shadow AI](https://customlabs.io/glossary/shadow-ai/) Glossary - [Prompt Injection](https://customlabs.io/glossary/prompt-injection/) Glossary - [Confused Deputy](https://customlabs.io/glossary/confused-deputy/) Glossary - [Data Processing Agreement (DPA)](https://customlabs.io/glossary/data-processing-agreement/) Glossary - [Data Residency](https://customlabs.io/glossary/data-residency/) Glossary - [Human Oversight](https://customlabs.io/glossary/human-oversight/) Glossary - [Canary Release](https://customlabs.io/glossary/canary-release/) Glossary - [Output Drift](https://customlabs.io/glossary/output-drift/) Glossary - [Model Deprecation](https://customlabs.io/glossary/model-deprecation/) Glossary - [System Card](https://customlabs.io/glossary/system-card/) Glossary ### 06 Cost - [What AI Actually Costs in Production, by Workload](https://customlabs.io/insights/ai-cost-benchmark-by-workload/) Insight - [Cutting Inference Spend 40% Without Betting on One Vendor](https://customlabs.io/case-studies/model-agnostic-routing-cut-inference-spend/) Case study - [Model cascade](https://customlabs.io/patterns/model-cascade/) Pattern - [Stable-prefix prompt caching](https://customlabs.io/patterns/stable-prefix-prompt-caching/) Pattern - [Retry-amplified spend](https://customlabs.io/failure-modes/retry-amplified-spend/) Failure mode - [Prompt cache never hits](https://customlabs.io/failure-modes/prompt-cache-never-hits/) Failure mode - [AI Cost Calculator](https://customlabs.io/tools/cost-calculator/) Tool - [CostMon](https://customlabs.io/products/costmon/) Page - [The AI Cost Model](https://customlabs.io/cost/) Page - [Inference Cost](https://customlabs.io/glossary/inference-cost/) Glossary - [Idempotency](https://customlabs.io/glossary/idempotency/) Glossary