How do you find out why one specific run went wrong?
One trace ID follows a single request across every hop it takes — retrieval, every model call, every tool call — logged with enough detail to reconstruct exactly what happened after the fact. A sampled slice of traces, weighted toward low-confidence or error-flagged runs, lands in a queue a human actually reviews on a schedule, not only when a complaint forces someone to go looking.
Also known as distributed tracing for agents, end-to-end trace ID, sampled human review queue
What problem does this solve?
Most AI systems accumulate logs — a retrieval log here, a model-call log there, a tool-call log somewhere else — with no shared identifier tying them to the same request. Reconstructing what happened on one bad run means manually correlating timestamps across systems that weren't built to be correlated, which is slow enough that most teams only do it once a user complaint forces the question.
Aggregate metrics — average latency, overall error rate — can look healthy while a specific, real failure sits unexamined in the tail, because nobody has a habit of looking at individual traces until something breaks loudly enough to demand it. By the time someone goes looking, the context needed to diagnose it, what exactly the model saw at each step, may be gone or hard to reassemble.
How does it work?
A single trace ID is generated at the start of each request and threaded through every subsequent hop — retrieval calls, every model invocation, every tool call — so the full sequence for one request can be pulled with one query, not reconstructed from scattered logs after the fact.
Each span in the trace records enough to actually diagnose a failure: the exact prompt sent, the exact response received, which tool was called with which arguments, and what came back, not just a duration and a status code. This is the raw material every other observability and eval practice depends on; without it, a "we can't reproduce it" conversation is the ceiling of what's diagnosable.
Rather than reviewing every trace, which is infeasible at volume, or none, which is where most teams default to, a sampling policy pulls a slice into a queue a human actually reviews on a schedule, weighted toward traces the system itself flagged as low-confidence, erroring, or unusually long, so the sample catches the cases most worth a human's attention, not a uniform random cross-section.
What are the moving parts?
- Generate one trace ID per request
Assign it at the entry point, before the first hop, so nothing downstream is uncorrelated.
- Thread the ID through every downstream hop
Carry it through every retrieval call, model call, and tool call for that request.
- Log the actual content per span
Record the real prompt, response, and tool arguments, not just timing and status.
- Weight sampling toward flagged runs
Favor low-confidence, error-flagged, or unusually long runs over a uniform random sample.
- Route the sample into a real human review queue
Send it to a queue with an owner and a schedule, not an unread archive.
- Feed findings back into evals or prompts
Close the loop by turning what review surfaces into new eval cases or prompt fixes.
When does it fit, and when doesn't it?
Use it when
- A support or engineering escalation currently requires manually correlating logs across two or more systems to explain one bad run
- The system has more than a couple of hops (retrieval + model + tool calls) where a failure could originate in any of them
- You want a repeatable way to catch drift or degradation before it's reported, not only after
- You are already building or maintaining a golden-set eval and need a source of real, hard cases to feed it
Don't use it when
- The system is a single model call with no tools or retrieval — there is no multi-hop sequence that needs correlating
- You have no plan or team capacity to actually staff the human review queue — collecting traces nobody reviews is storage, not observability
- Traffic volume is low enough that reviewing every single run by hand is already realistic — sampling infrastructure would be solving a problem you do not have yet
What does it actually cost?
A bad run becomes explainable from one query instead of a cross-system log correlation exercise
Storing full prompts, responses, and tool payloads per span raises storage volume and cost, especially at scale
A weighted sample surfaces the runs most worth human attention, not a random slice
The review queue needs a real, staffed owner and cadence, or the sampled traces just accumulate unread
Traces become the raw material for eval-set expansion and root-cause diagnosis alike
Full prompt/response logging raises data-handling and retention questions that need an explicit policy, not an afterthought
Drift and degradation become visible before a complaint forces the investigation
Threading one trace ID through every hop is real integration work across every service the request touches, not a config flag
How do you know it's working?
- Time-to-diagnose for a reported bad run, before and after adopting trace-first logging
- Sampling coverage — what share of low-confidence/error-flagged runs actually reach a human reviewer
- Review queue backlog and age, so a growing unreviewed backlog is caught early
- Count of review findings that fed back into the eval set or a prompt change
What failure modes does this prevent?
A Ship Audit checks which of these patterns your system actually needs, prioritized against what's most likely to break first.