Towards Self-Improving Error Diagnosis in Multi-Agent Systems

2026-04-19

AI Summary

This work addresses the challenge of debugging in multi-agent systems, where long interaction trajectories, inter-agent dependencies, and delayed error manifestation hinder precise identification of critical failure steps. To tackle this, the authors propose ErrorProbe, a semantic-level fault attribution framework operating in three stages: it first detects local anomalies through failure categorization, then performs symptom-driven backward tracing to prune irrelevant context, and finally employs a team of specialized agents—strategist, investigator, and arbiter—to collaboratively conduct tool-augmented verification. Notably, ErrorProbe requires no human annotations, constructs situation memories grounded solely in executable evidence, and enables cross-domain transfer without retraining. Evaluated on the TracerTraj and Who&When benchmarks, ErrorProbe substantially outperforms existing baselines, demonstrating particular strength in step-level error localization and cross-domain generalization.

Technology Category

• Multiagent Systems: Teamwork
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Large Language Model (LLM)-based Multi-Agent Systems (MAS) enable complex problem-solving but introduce significant debugging challenges, characterized by long interaction traces, inter-agent dependencies, and delayed error manifestation. Existing diagnostic approaches often rely on expensive expert annotation or ''LLM-as-a-judge'' paradigms, which struggle to pinpoint decisive error steps within extended contexts. In this paper, we introduce ErrorProbe, a self-improving framework for semantic failure attribution that identifies responsible agents and the originating error step. The framework operates via a three-stage pipeline: (1) operationalizing the MAS failure taxonomy to detect local anomalies, (2) performing symptom-driven backward tracing to prune irrelevant context, and (3) employing a specialized multi-agent team (Strategist, Investigator, Arbiter) to validate error hypotheses through tool-grounded execution. Crucially, ErrorProbe maintains a verified episodic memory that updates only when error patterns are confirmed by executable evidence, without the need for annotation. Experiments across the TracerTraj and Who&When benchmarks demonstrate that ErrorProbe significantly outperforms baselines, particularly in step-level localization, while the verified memory enables robust cross-domain transfer without retraining.

Problem

• Multi-Agent Systems
• Error Diagnosis
• Large Language Models
• Debugging
• Failure Attribution

Innovation

• self-improving diagnosis
• multi-agent systems
• error attribution
• tool-grounded execution
• episodic memory