Tracing Agentic Failure from the Flow of Success

2026-07-14

AI Summary

This work proposes an unsupervised failure attribution method for large language model agent systems that requires only successful execution trajectories for training, eliminating the need for costly prompts, failure examples, or step-level supervision. Framing the problem as a one-class learning task, the approach leverages neural controlled differential equations to model the dynamic patterns of successful trajectories in latent space and identifies failure steps through anomaly scoring. Experimental results demonstrate that with as few as 100 successful trajectories, the method achieves 200–5000× faster inference compared to existing approaches, while improving F1 scores by 20% in-domain and 7% out-of-domain. This represents the first lightweight, efficient, and failure-label-free solution for step-level failure attribution in agent systems.

Abstract

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.

Problem

• failure attribution
• LLM-based agentic systems
• unsupervised learning
• anomaly detection
• step-level error

Innovation

• unsupervised failure attribution
• one-class learning
• neural controlled differential equations
• agentic systems
• anomaly detection