CustomLabs
Search ⌘K Start a project →
Send a project brief
Library

Everything we've published.

143 entries across 11 sections: every insight, case study, comparison, pattern, failure mode, and reference page in one browsable index.

Every entry below has a Markdown twin. Add .md to its URL to read the plain version. See /llms.txt for the full rule.

01 Start here #

The twelve flagship guides. Read one front to back before diving into the reference material below.

The Applied AI Handbook

A six-stage handbook for shipping applied AI: decide, design, build, evaluate, operate, cost, each linked to the reference material behind it.

01 Decide

Should we build this with AI at all: build, buy, or skip?

7 min read

02 Design

What shape is the system: pipeline, agent, retrieval, or none of the above?

7 min read

03 Build

How do we get from a working notebook to a deployable service?

6 min read

04 Evaluate

How do we know it works, and how do we keep knowing after every change?

6 min read

05 Operate

What breaks in production, and how do we see it before the user does?

8 min read

06 Cost

What will this actually cost to run, and where does the spend hide?

6 min read

AI Security Review

What InfoSec, Privacy, Risk and Procurement ask before an AI feature ships. Each answer comes with the control and the evidence that pass the gate.

23 min read

The Agentic Delivery Playbook

The operating model for delivery with a fleet of coding agents. It has six control surfaces and 22 practices, plus the metrics that catch review debt early.

26 min read

The Eval Stack

How to know an AI system works before it ships. Read the eval layers, the checks behind them, and the ways an LLM judge lies.

21 min read

The Agent Tool Interface

Agents fail at the tool interface more often than at the model. The design rules for building an interface that holds up in production.

25 min read

Choosing an AI Delivery Partner

How to interrogate any AI delivery partner, us included, across every gate that matters, and the cases where we're the wrong call.

27 min read

The AI Cost Model

Where AI spend goes, the levers that move it, and the ways a cost estimate can be wrong before you commit budget.

27 min read

The AI Governance Layer

The standing regime that has to hold a year after the security review passes. Who is accountable, and how to prove the system behaves.

29 min read

The AI Release Path

How to ship a change to an AI system, ramp it safely, and reverse it fast if it turns out to be wrong.

34 min read

The Agent Adoption Playbook

The pilot worked. Here is how the rest of your engineers start using it, and the numbers that separate real adoption from seat activation.

33 min read

MCP in Production

MCPMCP is an open standard for connecting LLM applications to tools and data sources. standardizes the wire format. Identity, the catalog and the trust boundary are still yours to build, plus a ten-question gate for a new server.

42 min read

Context Engineering

What occupies a context windowThe context window is the maximum text, measured in tokens, a model can consider at once. at every step of a long run, and how to budget, compact, isolate and measure it before it fails.

38 min read

02 Architectures #

Six reference architectures for the systems we build. They sit one level above the patterns below.

All architectures →

A coding-agent delivery fleet

A control plane for running many coding agents at once. The agents do not collide on the same files. You can see what each one costs.

6 min read

03 Insights #

Dated articles on shipping applied AI. What breaks and how to design around it.

All insights →

Model-Agnostic by Design

Models change under you every few months: price, quality, and capability. So we never hardcode a single provider into a client's feature.

7 min read

Your RAG Demo Lied to You

Retrieval that looks flawless on ten clean PDFs falls apart on a real corpus. We explain why, and what a retrieval eval requires.

7 min read

What an AI Feature Costs in Production

Token costs that look trivial in a demo compound fast at scale. Make cost a first-class metric instead of a surprise on the invoice.

7 min read

04 Comparisons #

Decision guides for the architecture calls that come up on every engagement.

All comparisons →

RAG vs Fine-Tuning: Which One Solves Your Problem

Retrieval and fine-tuningFine-tuning trains a model's weights on your own examples, changing its behavior directly. solve different problems but get reached for interchangeably. How to tell which one your case needs, and why we default to retrieval.

7 min read

05 Case studies #

Engagement write-ups: the problem, what we built, and the measured outcome.

All case studies →

A Retrieval Pipeline That Cut Support Load

A SaaS support platform swapped a keyword search widget for a tenant-isolated retrieval pipeline, cutting escalations and response time with no new headcount.

6 min read

06 Patterns #

Named, opinionated patterns for building AI systems that survive production.

All patterns →

Bounded agent loop

A bounded agent loop runs under an explicit step budget and token ceiling.

5 min read

Structure-aware chunking

Chunk boundaries follow the document's own structure, such as headings and table rows.

5 min read

Golden-set gate in CI

A fixed set of human-labelled real cases runs in CI on every prompt or model change.

5 min read

Trace-first observability

One trace ID follows a single request across every hop, from retrieval to each model call and tool call.

5 min read

Model cascade

A cheap, fast model tries every request first.

5 min read

Stable-prefix prompt caching

Put invariant content first in the prompt and per-request content last, so the shared prefix stays byte-identical and the cache hits.

5 min read

07 Failure modes #

Concrete ways production AI systems break, each with symptoms, root cause, and a fix.

All failure modes →

Stale index serves deleted content

Your retrieval index was built once at ingest and never told the source changed. When a document is edited or deleted, nothing re-embeds the new version or tombstones the old chunk. The stale vector keeps scoring well and keeps getting served with confidence, giving the reader no signal that it is out of date.

4 min read

Chunk boundary splits the answer

The answer exists in the source, but a fixed-size chunker cut it in half at ingest time. A table row lost its header. A procedure lost its trigger condition. Each half scores weakly on its own, the ranker drops both, and retrieval reports nothing when the document plainly contains the answer.

4 min read

Similarity is not relevance

Cosine similarity rewards topical resemblance, not correctness. It can rank a document about the wrong product or the wrong date above the one that answers the query. It can do the same for the negated version of a claim. Embeddings capture topical similarity far more reliably than they capture identifiers, negation, or numbers.

4 min read

Unbounded agent loop

The agent has no step budget, no token budget, and no way to recognize it is stuck. A failing tool call stays in its context and keeps looking like a reasonable next try. The agent keeps trying variations of the same failed approach. Something external, such as a timeout or a human, finally stops it.

Tool argument hallucination

Loose tool schemas (free-form string IDs, everything optional) give the model room to fill a gap with something plausible-looking instead of something real. With no server-side validation catching the mismatch before execution, a confidently invented ID reaches a system that expects a real one.

4 min read

Silent tool failure

The tool returned HTTP 200 with an error message in the body, or an empty result set. The agent treated the absence of data as evidence and missed the failure. Nothing in the response forced a distinction between "nothing matched" and "something broke."

4 min read

Context overflow drops the task

As the conversation grows, a naive truncation strategy drops the oldest messages to stay under the context window. The oldest messages usually hold the system prompt and the original task state. The agent keeps running with no memory of what it was supposed to do.

4 min read

Vibes-based prompt regression

Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open. That is not a test. It's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.

Judge prefers its own output

A judge from the same model family as the generator tends to rate that family's output favorably. This is self-preference bias. A single vague rubric ('is this good?') collapses almost everything to a passing score. The eval suite then cannot tell a real regression from noise.

4 min read

Retry-amplified spend

The estimate priced the happy path: one clean call per outcome. Production reality includes retries on malformed or rate-limited calls. It includes fallbacks to a larger model when the first attempt fails. It also includes agent loops that make several calls per completed task. Every one of those multiplies calls per successful outcome without multiplying the original per-token estimate.

4 min read

Prompt cache never hits

A dynamic prefix changes the start of the prompt on every call. Examples are a timestamp, a per-user greeting, a reordered tool list, retrieved chunks placed before the static instructions. Prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front busts the whole cache.

4 min read

Injection via retrieved content

Yes. Retrieved content arrives on the same channel as instructions. A document, ticket, or webpage crafted or compromised to contain commands can make the model execute them with its real tool permissions. The system has no built-in way to tell 'instruction from us' apart from 'text we retrieved.'

4 min read

08 Tools & diagnostics #

Free, self-serve tools and fixed-scope diagnostics for scoping AI work before you commit budget.

All tools →

09 Topics #

Guides, architectures, comparisons, insights, case studies, patterns, failure modes and tools grouped by the six themes that recur across engagements.

All topics →

10 Glossary #

Plain-English definitions for the AI engineering vocabulary that matters.

All glossary terms →

11 The studio #

Who we are and how an engagement runs.

Source: https://customlabs.io/library/

↑↓ navigate ↵ select esc close