The question
Agent benchmarks preserve rich sequences of actions and observations, yet evaluation usually collapses each rollout into success, failure, or a scalar reward. That makes two agents with the same score look equivalent even when one avoids difficult regions and the other repeatedly enters them before recovering.
TraceGraph asks a process-level question: what does an agent encounter on the way to its outcome, and what does the benchmark require it to navigate?
From trajectories to a landscape
- Represent observable states. Each step becomes a sparse symbolic key set describing tool use, action intent, commands, file cues, observation patterns, temporal phase, and search behavior.
- Pool before identifying models. States from multiple models are combined for each task, then connected using IDF-weighted Jaccard similarity and mutual nearest neighbors.
- Recover process structure. Biconnected components form shared state regions; outcome diffusion adds productive cores, trap regions, failure basins, and recovery gates.
- Profile complete rollouts. Access, Trap exposure, and Repair summarize what each model supplies and what each benchmark rewards.
- Test a downstream use. On SWE-bench, graph-derived trap matches trigger lightweight continuation policies from the same prefix.
What the landscape reveals
The five evaluated benchmark splits do not reward the same navigation strategy. Some primarily reward avoiding trap regions; others distinguish agents that can repair after exposure. These differences are hidden by aggregate pass rates but become explicit in Access–Trap–Repair profiles.
The recovery experiment provides a concrete use case. On states where the detector fires, the best pooled single-factor policy improves official resolved rate from 40.4% to 43.5% on per-provider fired subsets and from 41.0% to 44.8% on common-fired instances. Provider-specific active components suggest that a single generic recovery instruction is not always optimal.
Interpretive boundaries
- The graph is a measurement of observable trajectory states, not a direct map of a model’s latent planning process.
- Productive cores and traps depend on the released rollout distribution and outcome overlay; they are not universal properties of a task.
- The intervention results apply to detector-fired subsets. They should not be read as an unconditional improvement across every SWE-bench instance.
- Symbolic signatures deliberately favor auditability, but may omit semantic distinctions that require richer representations.