How do you find out why one specific run went wrong?
One trace ID follows a single request across every hop, from retrieval to each model call and tool call. The system logs enough detail to rebuild what happened. A sampled slice of traces goes to a human review queue on a regular schedule. The sample favors low-confidence or error-flagged runs. Review starts long before a complaint forces a search.
Also known as distributed tracing for agents, end-to-end trace ID, sampled human review queue
What problem does this solve?#
Most AI systems collect logs in separate places. One log covers retrieval and another covers model calls. A third covers tool calls. No shared identifier ties them to the same request. To rebuild one bad run, an engineer must correlate timestamps by hand across systems that were never built to match. That is slow. Most teams do it only after a user complaint forces the question.
Aggregate metrics, such as average latency or overall error rate, can look healthy while a real failure sits unexamined in the tail. Nobody looks at individual traces until something breaks loudly. By then, the context needed to diagnose it may be gone or hard to reassemble. That context is what the model saw at each step.
How does it work?#
Generate a single trace ID at the start of each request. Thread it through every later hop. That includes retrieval calls and model calls. It also includes tool calls. One query then pulls the full sequence for a request. The team does not need to rebuild it from scattered logs.
Each span in the trace records enough to diagnose a failure. It stores the exact prompt sent and the exact response received. It also stores which tool the agent called, with which arguments, and what came back. A duration and a status code are too little. Every other observabilityObservability captures traces of every prompt, retrieval, tool call, and response for debugging. and eval practice depends on this raw material. Without it, "we can't reproduce it" is the limit of what the team can diagnose.
Reviewing every trace is infeasible at volume. Reviewing none is the default for most teams. A sampling policy pulls a slice into a queue, and a human reviews that queue on a schedule. The policy favors traces that the system flagged as low-confidence or erroring. The sample then holds the cases that most deserve human attention. A uniform random cross-section would miss many of them.
What are the moving parts?#
- Generate one trace ID per request
Assign it at the entry point, before the first hop, so every downstream step is correlated.
- Thread the ID through every downstream hop
Carry it through every call that request makes.
- Log the actual content per span
Record the real prompt and response, plus tool arguments. Timing and status are too little.
- Weight sampling toward flagged runs
Favor low-confidence or error-flagged runs over a uniform random sample.
- Route the sample into a human review queue
Send it to a queue that has an owner and a schedule. An unread archive does not count.
- Feed findings back into evalsAn eval suite is a versioned set of test cases that measures whether outputs are good. or prompts
Close the loop. Turn what review surfaces into new eval cases or prompt fixes.
When does it fit, and when doesn't it?#
Use it when
- A support or engineering escalation needs manual log correlation across two or more systems to explain one bad run
- The system has several hops, such as retrieval and model calls, and a failure could start in any of them
- You want a repeatable way to catch drift or degradation before a user reports it
- You build or maintain a golden-set eval and need a source of real, hard cases to feed it
Don't use it when
- The system is a single model call with no tools or retrieval. No multi-hop sequence needs correlation
- You have no plan or team capacity to staff the human review queue. Traces that nobody reviews are storage and give no observability
- Traffic is low enough that reviewing every run by hand is realistic. Sampling infrastructure would solve a problem you do not have yet
What does it cost?#
One query explains a bad run. The team no longer correlates logs across systems
Storing full prompts and responses, plus tool payloads, per span raises storage volume and cost, especially at scale
A weighted sample surfaces the runs that most deserve human attention. A random slice would miss many of them
The review queue needs a staffed owner and a cadence. Without them, sampled traces pile up unread
Traces give the team raw material for eval-set expansion and root-cause diagnosis
Full prompt and response logging raises data-handling and retention questions. The team needs an explicit policy up front
Drift and degradation become visible before a complaint forces an investigation
Threading one trace ID through every hop is integration work in each service the request touches. A config flag cannot do it
How do you know it's working?#
- Time-to-diagnose for a reported bad run, before and after adopting trace-first logging
- Sampling coverage: the share of low-confidence and error-flagged runs that reach a human reviewer
- Review queue backlog and age, so the team catches a growing unreviewed backlog early
- Count of review findings that fed back into the eval set or a prompt change
What failure modes does this prevent?#
Sources
- Google - Implementing SLOs
The method for setting a service-level objective our release scoreboard follows. Retrieved 2026-08-24.
- Google DORA - DORA Research
The research programme behind the delivery metrics our release and delivery pages measure. Retrieved 2026-08-24.
A Ship Audit checks which of these patterns your system needs. It ranks them by what is most likely to break first.