Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

2026-07-10

AI Summary

This work addresses the growing challenge of diagnosing increasingly subtle failures in AI agents by proposing Who&When Pro—the first large-scale benchmark for systematically evaluating the root-cause attribution capabilities of large language models (LLMs). The benchmark introduces a novel “replay-and-single-fault-injection” methodology, generating 12,326 multimodal failure trajectories with gold-standard labels across three modalities and 26 real-world scenarios. By strictly replaying successful execution traces and perturbing them with precisely localized faults, Who&When Pro enables rigorous assessment under a unified evaluation protocol and structured labeling scheme. Empirical analysis using this benchmark reveals systematic patterns in LLMs’ attribution performance across diverse modalities, model architectures, and interaction protocols, thereby establishing both a foundational dataset and methodological guidance for developing reliable automated attribution systems.

Abstract

Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.

Problem

• failure attribution
• AI agents
• LLMs
• agentic systems
• automated analysis

Innovation

• automated failure attribution
• large language models
• agentic systems
• benchmarking
• failure injection