OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

2026-06-25

AI Summary

Existing root cause analysis (RCA) datasets annotate only the final root cause, omitting causal propagation paths, which reduces evaluation to pattern matching and fails to assess large language models’ (LLMs’) genuine reasoning capabilities. To address this, this work introduces the PAVE protocol, which leverages intervention data from fault injection to reconstruct verifiable, step-by-step causal paths through a forward-validation mechanism, and presents OpenRCA 2.0—the first cross-system RCA benchmark with process-level annotations. By incorporating causal process supervision for the first time, experiments across 11 state-of-the-art LLMs reveal that only 20.7% of cases accurately recover the complete root cause set; while 76.0% identify at least one correct root cause service, merely 61.5% correctly embed it into a validated causal path, exposing a pervasive “ungrounded diagnosis” problem obscured by conventional evaluation metrics.

Abstract

Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.

Problem

• root cause analysis
• causal propagation path
• LLM agentic capabilities
• step-wise labeling
• outcome-only evaluation

Innovation

• causal propagation path
• step-wise labeling
• forward verification
• root cause analysis
• LLM agent evaluation