Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems

2025-04-30

AI Summary

This paper addresses the high human debugging cost and difficulty of failure attribution in LLM-based multi-agent systems after task failures. We formally define and pioneer the new research direction of *automated failure attribution*, aiming to precisely identify the faulty agent and the critical erroneous step(s). To this end, we construct *Who&When*, the first fine-grained annotated dataset comprising 127 real-world failure logs. Leveraging this dataset, we design three novel attribution methods that synergistically integrate LLM reasoning (e.g., o1, R1) with log analysis. Experimental results show that the best-performing method achieves 53.5% accuracy in agent-level attribution but only 14.2% in step-level localization—revealing a significant bottleneck for current SOTA models on this task. Our work establishes foundational data and methodology for advancing debuggability research in multi-agent LLM systems.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Failure attribution in LLM multi-agent systems-identifying the agent and step responsible for task failures-provides crucial clues for systems debugging but remains underexplored and labor-intensive. In this paper, we propose and formulate a new research area: automated failure attribution for LLM multi-agent systems. To support this initiative, we introduce the Who&When dataset, comprising extensive failure logs from 127 LLM multi-agent systems with fine-grained annotations linking failures to specific agents and decisive error steps. Using the Who&When, we develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons. The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps, with some methods performing below random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to achieve practical usability. These results highlight the task's complexity and the need for further research in this area. Code and dataset are available at https://github.com/mingyin1/Agents_Failure_Attribution

Problem

• Identifying responsible agents in LLM multi-agent system failures
• Automating failure attribution for debugging LLM agent systems
• Evaluating methods to pinpoint failure steps in agent tasks

Innovation

• Introduces automated failure attribution for LLM multi-agent systems
• Develops Who&When dataset with annotated failure logs
• Evaluates three methods for identifying failure causes