Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

2026-06-27

AI Summary

This study addresses a critical limitation in existing offline root cause analysis (RCA) benchmarks, which commonly rank methods using a single aggregated accuracy across heterogeneous subsystems—a practice that can mislead engineers into selecting suboptimal approaches for their specific subsystems. Through a comprehensive audit of three major benchmarks (OpenRCA, RCAEval, and PetShop), encompassing 11 subsystems and 778 evaluation instances, the authors employ random-effects modeling, 95% prediction intervals, case-level interaction tests, and a leave-one-subsystem-out cross-selection strategy to demonstrate that aggregated rankings obscure substantial and non-exchangeable performance variations at the subsystem level. In five out of six pairwise comparisons, the assumption of method exchangeability is rejected; furthermore, the leave-one-subsystem selection protocol selects a worse-performing method in up to five subsystems, incurring a maximum performance regret of 24.8 percentage points. The work advocates for fine-grained evaluation protocols and releases a 320-line reusable auditing module.

Abstract

Offline root-cause-analysis (RCA) benchmarks commonly rank methods by a single pooled top-1 accuracy across multiple subsystems, and engineers often read the pooled winner as a recommendation for their own subsystem. We audit that reading on three public RCA benchmark families -- OpenRCA, RCAEval, and PetShop -- covering 11 subsystems and 778 matched scoring units. To keep pairwise comparisons on identical cases, the main analysis retains four methods or comparators with complete coverage: BARO, a CD-1min adapter, max-$|Z|$, and per-service alert-count. All six pairwise comparisons show subsystem-level effects of both signs, every random-effects 95\% prediction interval crosses zero, and case-level interaction tests reject exchangeability in 5 of 6 pairs. Leave-one-system-out selection picks the lower-scoring method on up to 5 of 11 held-out subsystems, with regret reaching 24.8 pp on RCAEval / Sock-Shop. We release a 320-line audit module; given a matched RCA benchmark score table, it recomputes the same per-subsystem stability checks alongside pooled scores.

Problem

• root-cause analysis
• benchmarking
• pooled leaderboard
• subsystem-specific performance
• method evaluation

Innovation

• pooled leaderboard
• root-cause analysis
• benchmark audit
• subsystem-specific evaluation
• prediction interval