Symptom first, not theory first. Match what you're actually seeing to the failure mode that produces it, then confirm it before you fix anything. It's the same checklist a senior engineer runs during a Ship Audit.
| What you're seeing | Likely failure mode |
|---|---|
| A user reports an answer that cites a policy, price, or procedure the team is certain was changed or removed | Stale index serves deleted content |
| The cited source document, when you go look, no longer exists at the URL or path the answer referenced | Stale index serves deleted content |
| Two users ask the same question a week apart and get materially different answers with no index rebuild in between | Stale index serves deleted content |
| Support tickets mention "the bot is behind" more than once from unrelated teams | Stale index serves deleted content |
| A user pastes the exact sentence from the source document and retrieval still fails to surface the chunk containing it | Chunk boundary splits the answer |
| The answer is present when you open the document manually but absent from every retrieved chunk in the trace | Chunk boundary splits the answer |
| Failures cluster around tables, numbered procedures, and multi-clause definitions rather than plain prose | Chunk boundary splits the answer |
| Increasing top-k retrieved chunks barely moves the failure rate, because more of the same badly-cut chunks doesn't fix a chunk that's missing half its meaning | Chunk boundary splits the answer |
| Retrieval confidently returns a document about a similar-sounding product, plan tier, or account instead of the one the user actually asked about | Similarity is not relevance |
| A query containing a specific ID, date, or code returns topically related but factually wrong matches | Similarity is not relevance |
| A negated statement ("does not support X") and its positive counterpart ("supports X") both score highly against the same query | Similarity is not relevance |
| Keyword search on the same query would have found the right document immediately, but the vector search buried it | Similarity is not relevance |
| A run's duration or token spend is an order of magnitude above the median for the same task type | Unbounded agent loop |
| The trace shows the same tool called repeatedly with slightly varied arguments and no forward progress | Unbounded agent loop |
| The agent eventually times out or hits a platform limit rather than reaching a terminal state on its own | Unbounded agent loop |
| Cost per completed task has a long, expensive tail that the average masks | Unbounded agent loop |
| Tool calls fail with a 404 or "not found" at a noticeably higher rate than you would expect from legitimate user error | Tool argument hallucination |
| A rejected call's argument looks like a real ID in shape (right format, right length) but doesn't correspond to anything in the system | Tool argument hallucination |
| The invalid ID doesn't appear anywhere earlier in the conversation or in any prior tool result the agent received | Tool argument hallucination |
| The failure rate rises specifically on tasks where the agent must recall an identifier rather than one just handed to it in a prior step | Tool argument hallucination |
| The agent states something as fact ("there are no open orders") immediately downstream of a tool call that actually errored | Silent tool failure |
| Logs show a 200-status response with an error string embedded in the body, treated by the agent as normal data | Silent tool failure |
| An empty list comes back from a tool and the agent proceeds as if that confirms absence rather than possible failure | Silent tool failure |
| Per-tool error rates in your metrics are near zero while user-reported wrongness for the same tool is not | Silent tool failure |
| An agent behaves correctly for the first several steps of a long session, then starts ignoring constraints stated at the very start | Context overflow drops the task |
| The transcript, inspected at step 20, no longer contains the original system prompt or task specification | Context overflow drops the task |
| Behavior at step 1 and behavior at step 20 diverge on rules that were never explicitly rescinded mid-session | Context overflow drops the task |
| Runs crossing a certain length or turn count show a step change in error rate rather than a gradual decline | Context overflow drops the task |
| A prompt or model change ships after "looking good" in a manual spot-check, with no fixed test set run before or after | Vibes-based prompt regression |
| Quality complaints rise after a change that the team considered minor or purely stylistic | Vibes-based prompt regression |
| Nobody can answer "did this get better or worse" with a number — only with an impression | Vibes-based prompt regression |
| The same class of failure gets fixed, ships fine for a while, then quietly reappears after an unrelated later change | Vibes-based prompt regression |
| Nearly every case in the eval suite scores at or near the top of the scale, regardless of known quality differences between them | Judge prefers its own output |
| A deliberately degraded output (injected typo, wrong fact, missing field) still passes the judge's review | Judge prefers its own output |
| The judge and the generator are the same model, or from the same provider and generation | Judge prefers its own output |
| Human reviewers, sampled on the same cases, disagree with the judge meaningfully more often than the team expected | Judge prefers its own output |
| Actual monthly inference spend runs meaningfully above the number modeled from expected traffic times per-token price | Retry-amplified spend |
| The gap between sticker price and actual spend grows specifically on features that retry, fall back, or chain multiple calls | Retry-amplified spend |
| Nobody can currently answer "what is our cost per successful outcome" without manually reconstructing it from raw billing data | Retry-amplified spend |
| A recent quality-improving change (bigger model, more context, self-correction pass) coincided with a step change in spend that wasn't separately budgeted | Retry-amplified spend |
| Provider usage data shows near-zero cache-read tokens relative to total input tokens, despite sending largely similar prompts repeatedly | Prompt cache never hits |
| The system prompt and tool definitions are effectively static, yet caching discounts aren't showing up in the bill | Prompt cache never hits |
| Something like a timestamp, request ID, or session-specific greeting appears near the start of the prompt rather than the end | Prompt cache never hits |
| Retrieved or dynamically ordered content (RAG chunks, tool lists) is placed before the static instructions in the prompt template | Prompt cache never hits |
| An agent takes an action that traces back to text inside a retrieved document, ticket, or webpage rather than to the user's actual request | Injection via retrieved content |
| A tool call happens that doesn't correspond to anything the user asked for, immediately following a retrieval or content-fetch step | Injection via retrieved content |
| A document in the corpus contains phrasing that reads like an instruction ("system note:", "ignore the above", "process this automatically") rather than like content | Injection via retrieved content |
| A red-team test document planted in the corpus successfully triggers an unintended action when it's retrieved | Injection via retrieved content |
Your retrieval index was built once at ingest and never told the source changed. When a document is edited or deleted, nothing re-embeds the new version or tombstones the old chunk, so the stale vector keeps scoring well and keeps getting served — confidently, and with no signal to the reader that it is out of date.
Read the diagnosis →The answer exists in the source, but a fixed-size chunker cut it in half at ingest time — a table row split from its header, a procedure split from its trigger condition. Each half scores weakly on its own, the ranker drops both, and retrieval reports nothing when the document plainly contains the answer.
Read the diagnosis →Cosine similarity rewards topical resemblance, not correctness — it can rank a document about the wrong product, the wrong date, or the negated version of a claim above the one that actually answers the query, because embeddings represent "about the same thing" far more reliably than they represent identifiers, negation, or numbers.
Read the diagnosis →The agent has no step budget, no token budget, and no way to recognize it is stuck — a failing tool call stays in its context and keeps looking like a reasonable next thing to try, so it keeps trying variations of the same failed approach until something external (a timeout, a bill, a human) stops it.
Read the diagnosis →Loose tool schemas — free-form string IDs, everything optional — give the model room to fill a gap with something plausible-looking instead of something real, and with no server-side validation catching the mismatch before execution, a confidently invented ID reaches a system that expects a real one.
Read the diagnosis →The tool returned HTTP 200 with an error message in the body, or an empty result set, and the agent read the absence of data as evidence rather than as a failure — because nothing in the response forced a distinction between "nothing matched" and "something broke."
Read the diagnosis →As the conversation grows, a naive truncation strategy drops the oldest messages to stay under the context window — and the oldest messages are exactly where the system prompt and the original task state usually live, so the agent keeps running with no memory of what it was actually supposed to do.
Read the diagnosis →Without a labelled eval set, the change was graded against whatever two or three examples the author happened to have open — which is not a test, it's an anecdote. A regression anywhere outside that narrow, unrepresentative sample ships straight to production undetected.
Read the diagnosis →A judge from the same model family as the generator tends to rate that family's output favorably — self-preference bias — and a single vague rubric ('is this good?') collapses almost everything to a passing score, so the eval suite stops being able to tell a real regression from noise.
Read the diagnosis →The estimate priced the happy path — one clean call per outcome. Production reality includes retries on malformed or rate-limited calls, fallbacks to a larger model when the first attempt fails, and agent loops that make several calls per completed task, and every one of those multiplies calls per successful outcome without multiplying the original per-token estimate.
Read the diagnosis →A dynamic prefix — a timestamp, a per-user greeting, a reordered tool list, retrieved chunks placed before the static instructions — changes the start of the prompt on every call, and prompt caching only pays off when the shared prefix is byte-identical across requests. One volatile token near the front is enough to bust the whole cache.
Read the diagnosis →Yes — retrieved content arrives on the same channel as instructions, so a document, ticket, or webpage crafted (or compromised) to contain commands can have the model execute them with its real tool permissions, and the system has no built-in way to tell 'instruction from us' apart from 'text we retrieved.'
Read the diagnosis →No failure modes match that filter yet.
The questions that come up before someone decides whether to fix this in-house or bring in a second set of eyes.
Start from the symptom, not the theory. The table above maps observable symptoms (what a user reported, what a trace shows) to the specific failure mode that produces them. Match your symptom first, then read that entry's "how do you confirm it's this" section to verify before you start fixing anything.
Yes. Every entry here includes the actual checks we run: grep the corpus, diff a trace, measure a ratio. Each is written so you can run it yourself this week. Start with the category your symptom falls into, work through detection before jumping to a fix, and only escalate if the checks come back ambiguous.
This page is the checklist; a Ship Audit is a senior engineer running it against your specific system, with the context to spot which failure modes are actually live versus theoretical, and a written, prioritized plan for fixing what's found. It's the same diagnostic discipline, applied by people who've seen the edge cases before.
This page is the checklist we audit against. A Ship Audit is us running it against your system, with a written, prioritized plan for what we find.