AI Agent Debugging: Your Dashboard Knows the Cost and Not the Cause

AI agent debugging starts where your observability dashboard ends. When an AI coding agent breaks something and the run is already over, the dashboard can tell you what the run cost (tokens, dollars, latency), and it will be silent on the only question you actually have: why did the agent change that file? The artifact that answers it is a recorded trace you can open, read, and replay, because it hands the run back to you instead of describing it from the outside.

This stopped being a rare event the moment agents started doing real work. An agent will sweep through a repository, edit several files, run the checks, and report success, all between two prompts you typed minutes apart. If one of those edits is wrong, you find out later: after the terminal is closed, after the scrollback is gone, after the process that could have explained itself has exited. What happens next depends entirely on what you kept. If the answer is a cost dashboard, you are about to do archaeology. If the answer is a recording, you are about to do reading.

The failure you cannot reproduce

Here is the shape of it. You come back to the repo and a file you never asked anyone to touch has been rewritten, or deleted, or emptied of the function everything else depends on. You ask the agent what happened; the session is closed, and even where a transcript survives, the agent's account of its own run is a reconstruction, not a recording. So you do the natural thing and run it again, and you get a different run. Different tool calls, different edits, possibly no failure at all, because the original trajectory depended on sampling, on the state of the repo, on timing. The run you need to inspect does not exist anymore.

It is worse than unreproducible. It is unreproducible and marked as success. A run exits 0 when the agent exits 0, even if a check inside the run exited 1: the pipeline can be green while a verification step inside the run failed, and the exit code you would naturally trust is telling you nothing.

It does not matter which model the router picked for the run (GLM 5.3 Flash or anything else): once the process exits, the reasoning is gone with it. The evidence existed only while the run was live: the prompts, the tool calls, the outputs, the diffs. If nothing recorded them, "why did it change that file" has no answer. It has theories.

This is the failure mode that separates AI coding agents from every tool that came before them: the damage and the explanation happen in the same place, and the place closes.

What does a dashboard measure, and what does it step over?

The instinct after a bad run is to open the observability dashboard, and the dashboard will be genuinely good at its job. Its job is traffic: tokens per day, cost per model, latency, error rates. For capacity planning and billing that is exactly the right instrument, and if you run agents in production you should have it open.

But your question is not aggregate. It is singular and causal: why did this run change this file? Aggregation steps over precisely the granularity that answers it. Averaged across runs, the run you care about is noise; inside that run, the tool call you care about is noise again.

A dashboard describes a run from the outside: that it happened, what it weighed, what it cost. It cannot hand you the run, and "why" is not a property of the description. It is a property of the sequence.

• What did the run cost? — Cost dashboard Answers it vs Recorded trace Answers it

• Why did the agent change that file? — Cost dashboard No answer vs Recorded trace The edit, in sequence, with its diff

• Which check failed inside a green run? — Cost dashboard No answer vs Recorded trace The check, with its exit code

• Can I run the exact failure again? — Cost dashboard No vs Recorded trace Yes, offline, at no cost

The layer that answers it sits underneath: recorded request logs, captured as the run happened, with every prompt sent, every tool call issued, every response that came back, in order. Not a summary of the run. The run itself.

Reading one run as a timeline

With a recording, debugging stops being archaeology and becomes reading. Archaeology is what you do without one: git reflog, stash entries, shell history, your own recollection of what you asked for earlier that day. Reading is what you do with one: open the timeline and scroll.

The timeline lays the run out in the order it happened: the prompt that started it, each tool call, each file edit with its diff, each check, each exit code. A filesystem snapshot is taken once per turn rather than once per tool call, which is enough to see the repo's state at every step of the conversation without drowning in per-call noise. What makes this debugging rather than browsing is adjacency: the edit and the check that caught it sit next to each other, in order, with nothing in between to speculate about. "Why" is mostly a property of adjacency.

A concrete example, read off a recorded fix: 14 events, including the file change printed as +1 -3 and a failing check at exit 1. The edit, then the check that failed because of it, adjacent in the record. That is the entire difference between reconstructing a run from fragments and reading one. It matters most for terminal coding agents, whose workspace is a terminal that closes the moment the job is done: the timeline is the scrollback that survives.

Recorded or inferred: what the trace knows versus what it worked out

A timeline tells you what happened in order. The causal graph tells you what led to what, and the gap between those two is where trust has to be earned.

The graph connects events: this edit, then this failing check. Some of those edges are recorded facts: the tool call that produced the diff is right there in the trace. Others are inferred: the graph's conclusion that the check failed because of that diff. orca graph labels every edge recorded or inferred and names the rule it used either way, so you always know whether you are looking at something the run did or something the tool worked out about the run.

That distinction is enforced, not aspirational: inferred edges are never written back into the trace. The trace remains a faithful record of what happened; inference is a view on top of it, which you can inspect, question, and disagree with. This matters most when more than one agent is involved. When a refactor agent and a test-writing agent touch the same files, "which agent caused this" is precisely the question multi-agent attribution exists to answer. An edge that quietly promotes itself from inference to fact is how you end up debugging a story instead of a run.

Reproducing it as often as you like for nothing

Reading explains. Replay proves. Once you have a hypothesis (the check failed because the edit removed the reset call), you want to run it again and watch it happen. Rerunning the live agent buys you a new trajectory and a new bill.

Replaying the recording buys you the same run: replay runs with the network blocked, so it costs no tokens and has no variance. Same events, every time, offline. That is the property that turns agent debugging from gambling into engineering: the failure has become deterministic, and deterministic failures get fixed.

The tooling is not a black box, either. OrcaReplay is open source under Apache-2.0 and the trace format is CC BY 4.0, so anyone may reimplement it: your recordings are not hostage to a proprietary format, ours included. And it is exercised, not demoed: 1393 tests on Node 20 and Node 22. You can read the source, check the format, and run the suite yourself before you trust any of it with your team's runs.

The takeaway

A dashboard is a bill. A recorded trace is the run. If your plan for debugging AI agents ends at a cost dashboard, you do not have a debugging plan: you have a billing system. The dashboard will always be able to tell you what a run cost, and it will never be able to tell you why the agent deleted your file, because "why" lives in the sequence, and the sequence exists only if you kept it.

The whole method is four steps:

• Record the runs.

• Read the timeline.

• Check the graph's edges.

• Replay the ones that frighten you, for free, as often as you like.

Sourcing note: Every figure in this article is vendor-reported, from our own product and from the OrcaReplay repository and its documentation: the exit-code behavior of a run, the 14-event recorded fix with its +1 -3 diff and its exit-1 check, the edge labeling in orca graph, the once-per-turn snapshot cadence, offline replay with the network blocked, and the 1393-test suite on Node 20 and Node 22. No third-party measurements are cited anywhere in this article. The 1393-test suite is the one claim you can verify yourself, by cloning the repository and running it. All items were last checked on 2026-09-04.