Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

2026-07-08

AI Summary

In large language model (LLM)-based multi-agent systems, prolonged interactions and behavioral coupling render it challenging to identify the responsible agent and the earliest critical failure step following a system breakdown. To address this issue, this work proposes AgentLocate, a framework that integrates an LLM-based adjudicator with multi-perspective independent evaluations. It employs a confidence-aware strategy to aggregate judgments and incorporates lightweight fine-tuning to enhance adjudication performance. AgentLocate achieves, for the first time, joint and precise attribution of both the culpable agent and the decisive failure step. Evaluated on two diverse benchmarks, the method significantly outperforms existing approaches while maintaining high accuracy and low computational overhead.

Abstract

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Problem

• failure localization
• LLM-based multi-agent systems
• agent attribution
• trajectory analysis
• system diagnosis

Innovation

• failure localization
• multi-agent systems
• LLM-based reasoning
• confidence-aware aggregation
• lightweight fine-tuning