Problem
Research questions and friction points this paper is trying to address.
agent failure diagnosis
long-horizon tasks
context window limitation
fault attribution
autonomous agents
SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
🤖 AI Summary
Current large language models struggle to diagnose failures in extremely long multi-agent trajectories due to inherent context length limitations. This work proposes a tool-augmented active diagnostic loop framework that enables the model to actively focus on critical trajectory segments through chunked retrieval, short-term memory mechanisms, and cross-turn reasoning, effectively decoupling diagnostic accuracy from context window constraints. By introducing an active investigation paradigm into agent failure attribution for the first time, the method achieves state-of-the-art performance, surpassing existing approaches by 20% on the Who&When dataset (1M tokens) and by 19% on the TRAIL GAIA subset (25K tokens). Notably, it maintains a precision of 0.58 even when the failure location lies five times beyond the model’s native context length.
📝 Abstract
As autonomous agents tackle increasingly complex multi-step, multi-agent tasks, their execution trajectories have scaled beyond the constraints of even the largest context windows. Current methods for effectively diagnosing agent failures load the full trajectory into an LLM's context window, which suffers from attention dilution and fails when agentic traces inevitably exceed context limits. To address this, we introduce SAFARI (Scaling long-horizon Agentic Fault AttRibution via active Investigation), a framework that replaces linear context loading with a tool-augmented diagnostic loop. By equipping LLMs with a specialized toolbox to read and search trajectory segments alongside a persistent Short-Term Memory (STM) for cross-turn reasoning, SAFARI effectively decouples diagnostic accuracy from architectural context limits. Our experiments demonstrate that SAFARI outperforms state-of-the-art results by 20% on the Who&When dataset within a 1M token budget, and by 19% on TRAIL GAIA subset on a 25K token budget. Most significantly, SAFARI maintains a 0.58 precision even when the target fault resides 5x beyond the model's native context window, a scenario where traditional evaluators fail entirely.
Innovation
Methods, ideas, or system contributions that make the work stand out.
active investigation
tool-augmented reasoning
short-term memory
long-horizon fault attribution
context window decoupling
🔎 Similar Papers
No similar papers found.
