CustomLabs
Search ⌘K Start a project →
Send a project brief
Library

Everything we've published.

109 entries across 10 sections: every insight, case study, comparison, pattern, failure mode, and reference page in one browsable index.

01 Start here

The five flagship guides — read one front to back before diving into the reference material below.

The Applied AI Handbook

A six-stage handbook for shipping applied AI — decide, design, build, evaluate, operate, cost — mapping the studio's entire knowledge base onto the decisions a real project actually meets.

01 Decide

Should we build this with AI at all — build, buy, or skip?

02 Design

What shape is the system: pipeline, agent, retrieval, or none of the above?

03 Build

How do we get from a working notebook to a deployable service?

04 Evaluate

How do we know it works, and how do we keep knowing after every change?

05 Operate

What breaks in production, and how do we see it before the user does?

06 Cost

What will this actually cost to run, and where does the spend hide?

AI Security Review

The questions InfoSec, Privacy, Risk, and Procurement ask before an AI feature ships — what passes, the control, and the evidence you hand over.

The Agentic Delivery Playbook

The operating model for running real software delivery with a fleet of coding agents, first-hand from the studio's own delivery system.

The Eval Stack

Six eval layers, 24 named checks, five ways an LLM judge lies, and the numbers worth putting on a dashboard — how to know an AI system actually works before it ships.

The Agent Tool Interface

Six interface surfaces, 24 named design rules, and five named failures — how to design the tools and context an agent actually works through.

02 Insights

Dated articles on shipping applied AI — what breaks, what it costs, and how to design around it.

All insights →

Model-Agnostic by Design

Models change under you every few months: price, quality, and capability. Here's why we never hardcode a single provider into a client's feature.

Your RAG Demo Lied to You

Retrieval that looks flawless on ten clean PDFs falls apart on a real corpus. Here's why, and what evaluating retrieval quality actually requires.

03 Comparisons

Decision guides for the architecture calls that come up on every engagement.

All comparisons →

04 Case studies

Engagement write-ups — the problem, what we built, and the measured outcome.

All case studies →

The AI-Native Target That Wasn't: A Two-Week Diligence

A growth-equity firm had a term sheet out on an 'AI-native' SaaS target. A two-week technical readiness assessment surfaced $1.4M of undisclosed re-platform cost and a vendor dependency that would triple at projected volume, and repriced the deal.

Cutting Inference Spend 40% Without Betting on One Vendor

A Series B fintech was scaling LLM spend faster than revenue on a single provider. A model-agnostic routing layer cut unit cost, killed vendor concentration risk, and met a data-residency requirement its incumbent couldn't.

Retrieval Pipeline That Actually Cut Support Load

A mid-market SaaS support platform replaced a keyword search widget with a tenant-isolated retrieval pipeline, cutting escalations and first-response time without a headcount increase.

05 Patterns

Named, opinionated patterns for building AI systems that survive production.

All patterns →

Intent router to specialists

A cheap, fast classifier reads the incoming request first and routes it to one of several narrow, single-purpose agents — each holding only the tools, context, and instructions its job needs — instead of a single god-agent carrying every tool definition and every rule for every possible request.

Bounded agent loop

An agent loop runs under an explicit budget — a maximum step count, a token ceiling, and a wall-clock limit — plus a termination contract that forces every run to end in one of a small number of named states: success, failure, or escalation.

Structure-aware chunking

Chunk boundaries follow the document's own structure — headings, table rows, list items, section boundaries — instead of a fixed token count, and each chunk carries its parent heading or identifying context in its own body.

Retrieve-then-rerank

A cheap, high-recall first pass — vector search, optionally fused with keyword search — pulls a wide candidate set of 50 to 100 documents likely to contain the right answer somewhere.

Change-data-capture ingest

The ingest pipeline subscribes to the actual change events of the source system — a webhook, a CMS publish hook, a database trigger — instead of re-crawling on a fixed schedule.

Typed tool contract

Every tool argument is defined by a strict JSON schema — enums for known value sets, validated patterns for IDs, required fields wherever the tool genuinely needs them — with no free-text catch-all surface.

Human checkpoint before irreversible actions

Every tool the agent can call is scoped to the narrowest permission the task genuinely needs, and any action that can't be cleanly undone — a refund, a delete, an external message — requires an explicit human confirmation before it executes, not just a plausible-looking model decision.

Golden-set gate in CI

A fixed, human-labelled set of real cases — each with a specific, checkable expected property, not a vibe — runs automatically in CI on every prompt or model change.

Trace-first observability

One trace ID follows a single request across every hop it takes — retrieval, every model call, every tool call — logged with enough detail to reconstruct exactly what happened after the fact.

Model cascade

A cheap, fast model attempts every request first.

Stable-prefix prompt caching

The prompt is ordered with everything invariant across calls first — system instructions, tool definitions in a fixed serialization order, few-shot examples — and everything that changes per request — retrieved chunks, user input, timestamps — placed last.

06 Failure modes

Concrete ways production AI systems break, each with symptoms, root cause, and a fix.

All failure modes →

Stale index serves deleted content

Your retrieval index was built once at ingest and never told the source changed. When a document is edited or deleted, nothing re-embeds the new version or tombstones the old chunk, so the stale vector keeps scoring well and keeps getting served — confidently, and with no signal to the reader that it is out of date.

Chunk boundary splits the answer

The answer exists in the source, but a fixed-size chunker cut it in half at ingest time — a table row split from its header, a procedure split from its trigger condition. Each half scores weakly on its own, the ranker drops both, and retrieval reports nothing when the document plainly contains the answer.

Similarity is not relevance

Cosine similarity rewards topical resemblance, not correctness — it can rank a document about the wrong product, the wrong date, or the negated version of a claim above the one that actually answers the query, because embeddings represent "about the same thing" far more reliably than they represent identifiers, negation, or numbers.

Unbounded agent loop

The agent has no step budget, no token budget, and no way to recognize it is stuck — a failing tool call stays in its context and keeps looking like a reasonable next thing to try, so it keeps trying variations of the same failed approach until something external (a timeout, a bill, a human) stops it.

Tool argument hallucination

Loose tool schemas — free-form string IDs, everything optional — give the model room to fill a gap with something plausible-looking instead of something real, and with no server-side validation catching the mismatch before execution, a confidently invented ID reaches a system that expects a real one.

Silent tool failure

The tool returned HTTP 200 with an error message in the body, or an empty result set, and the agent read the absence of data as evidence rather than as a failure — because nothing in the response forced a distinction between "nothing matched" and "something broke."

Context overflow drops the task

As the conversation grows, a naive truncation strategy drops the oldest messages to stay under the context window — and the oldest messages are exactly where the system prompt and the original task state usually live, so the agent keeps running with no memory of what it was actually supposed to do.

Vibes-based prompt regression

Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open — which is not a test, it's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.

Judge prefers its own output

A judge from the same model family as the generator tends to rate that family's output favorably — self-preference bias — and a single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise.

Retry-amplified spend

The estimate priced the happy path — one clean call per outcome. Production reality includes retries on malformed or rate-limited calls, fallbacks to a larger model when the first attempt fails, and agent loops that make several calls per completed task, and every one of those multiplies calls per successful outcome without multiplying the original per-token estimate.

Prompt cache never hits

A dynamic prefix — a timestamp, a per-user greeting, a reordered tool list, retrieved chunks placed before the static instructions — changes the start of the prompt on every call, and prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front is enough to bust the whole cache.

Injection via retrieved content

Yes — retrieved content arrives on the same channel as instructions, so a document, ticket, or webpage crafted (or compromised) to contain commands can have the model execute them with its real tool permissions, and the system has no built-in way to tell 'instruction from us' apart from 'text we retrieved.'

07 Tools & diagnostics

Free, self-serve tools and fixed-scope diagnostics for scoping AI work before you commit budget.

All tools →

AI Cost Calculator

What AI actually costs to run in production — not the sticker price.

08 Topics

Insights and case studies grouped by the six themes that recur across engagements.

All topics →

09 Glossary

Plain-English definitions for the AI engineering vocabulary that matters.

All glossary terms →

10 The studio

Who we are, what we build, and how an engagement runs.

↑↓ navigate ↵ select esc close