root cause and failure analysis

Root cause analysis uses instrumentation, logs, traces, postmortems, and causal-debugging (span correlation, dependency graphs, statistical debugging) to locate and diagnose system failures, reproduce incidents, and recommend fixes; practitioners use tools like Sentry, Jaeger, Prometheus, and structured RCA methods (5 whys, fishbone).

rootcauseandfailure

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

To address the challenges of root cause localization in complex software systems—particularly susceptibility to spurious correlations and incomplete domain expertise—this paper proposes a causal graph modeling method that integrates partial domain knowledge. We introduce a novel four-stage framework: (1) initial causal structure learning via PC/GES variants; (2) reliability enhancement of causal edges using graph neural networks; (3) redundancy elimination through counterfactual reasoning; and (4) lightweight domain knowledge injection, enabling analysts to initiate analysis with only localized expert priors. Evaluated on both synthetic and real-world industrial datasets, our approach achieves a 27.3% improvement in root cause localization accuracy and reduces average causal path length by 41%, outperforming state-of-the-art causal discovery and correlation-based methods. The framework has been deployed in a cloud platform’s performance operations system.

Causal InferenceComplex SystemsPerformance Troubleshooting

Root cause localization in complex, dynamic multi-layer business systems suffers from poor interpretability and difficulty in tracing multi-hop causal dependencies. Method: This paper proposes an end-to-end causal inference framework that uniquely integrates conditional anomaly scoring, counterfactual noise attribution, and depth-first graph search—implemented atop DoWhy—to enable interpretable, backward tracing of multi-hop causal paths from observed anomalies to their initial triggers. Unlike conventional correlation- or rule-based approaches, it reconstructs the full causal chain rather than identifying isolated correlations. Contribution/Results: Evaluated on synthetic anomaly injection benchmarks, the framework achieves significantly higher root cause ranking accuracy than state-of-the-art baselines. It supports actionable root cause diagnosis in dynamic environments by delivering both precise causal attribution and human-interpretable explanations grounded in structural causal models.

Addresses limitations of traditional RCA methods in complex systemsDevelops a causal inference package for actionable root cause analysisIdentifies and ranks root causes by tracing multi-hop causal chains

This work addresses the limitations of existing data-driven root cause analysis methods, which rely on the causal sufficiency assumption and suffer significant performance degradation in partially observable systems with unmeasured latent variables. The authors propose a novel approach based on partial ancestral graphs (PAGs), modeling system failures as parametric interventions and integrating causal effect identification with partial identification theory to rank candidate root causes. Notably, for non-identifiable scenarios, the method introduces analytical causal bounds for the first time in root cause analysis. This framework is the first to jointly handle latent variables and partial identifiability, thereby eliminating dependence on causal sufficiency. Experiments demonstrate that the proposed method substantially outperforms state-of-the-art approaches across synthetic data, microservice anomaly benchmarks, and power grid cascading failure datasets, confirming its robustness and effectiveness in complex, partially observable environments.

Causal SufficiencyLatent ConfoundersPartial Ancestral Graphs

Root Cause Analysis of Outliers with Missing Structural Knowledge

Jun 07, 2024
NO
Nastaran Okati
🏛️ Max Planck Institute for Software Systems | Max Planck Institute for Intelligent Systems | University of Cambridge | Amazon Research

Real-world root cause analysis (RCA) faces a critical challenge: post-intervention distributions often contain only a few—or even a single—sample, rendering distribution-dependent or low-density-region regression methods statistically ill-posed. This paper proposes a lightweight root cause identification framework that requires neither counterfactual reasoning nor a fully specified structural causal model (SCM). It operates either given a causal DAG or, in the absence of one, solely from an anomaly score ranking. We theoretically prove that low-scoring anomalies rarely trigger high-scoring ones and derive a probabilistic upper bound on non-monotonic propagation paths. By abandoning Shapley-value-based attribution and density-sensitive regression, our method achieves linear time complexity O(n). It eliminates SCM fitting and counterfactual computation while providing rigorous theoretical guarantees and strong empirical performance.

Addresses single-sample limitations in post-intervention distribution analysisIdentifies root causes of anomalies with missing causal graph knowledgeProvides guarantees for root cause detection in polytree structures

To address the limitations of unimodal modeling and offline analysis in online multivariate microservice root-cause identification, this paper proposes OCEAN—the first online multivariate causal graph learning framework. OCEAN integrates dilated convolutional neural networks with graph neural networks, and introduces two key innovations: a multifactor attention mechanism and a contrastive mutual information maximization module for graph fusion. These components jointly enable real-time causal structure learning and dynamic cross-modal relational modeling over heterogeneous telemetry—such as metrics and logs. Evaluated on three real-world industrial datasets, OCEAN achieves an average 18.7% improvement in root-cause localization F1-score while maintaining inference latency under 200 ms. The framework thus delivers a scalable, production-deployable paradigm that simultaneously ensures high accuracy and strict real-time constraints for online multivariate fault attribution.

Learning causal structures from diverse system data streamsOnline multi-modal root cause analysis in microservicesOvercoming computational and single-modal limitations in RCA

Latest Papers

What's happening recently
View more

This work addresses a critical limitation in existing root cause analysis methods for anomalies: their failure to distinguish between two fundamentally distinct sources—measurement errors and mechanism shifts—often leading to misdiagnosis. To resolve this, the paper proposes the first causal framework that explicitly models both anomaly types by treating them as implicit interventions on latent “true” variables and observed “measured” variables. A structural causal model (SCM) with latent variables is constructed, and maximum likelihood estimation is employed to simultaneously classify anomaly types and localize root causes. Theoretically, the approach is shown to be identifiable without requiring prior knowledge of the causal graph structure. Empirical evaluations demonstrate state-of-the-art performance in root cause localization, accurate anomaly-type classification, and robustness even when the underlying causal graph is unknown.

anomaly classificationcausal characterizationmeasurement anomalies

This work addresses the challenge of root cause localization in deployed multi-agent systems, where failures are often obscured by cascading effects, implicit dependencies, and long execution traces. To tackle this, the authors propose a lightweight causal tracing framework that, for the first time, applies causal graph reasoning to post-deployment fault diagnosis in such systems. The approach constructs a causal graph from execution logs and employs a backward-tracing algorithm combined with structural and positional features to rank candidate root causes—without relying on large language models. Experimental results demonstrate that the method achieves high-precision root cause identification across diverse failure benchmarks, with latency under one second, significantly outperforming both heuristic and LLM-based baselines while offering strong efficiency and interpretability.

causal tracingdeployed AI systemsfailure diagnosis

This work addresses the limited interpretability and accountability of large language models (LLMs) in root cause analysis, which hinder their applicability in high-stakes operational settings requiring rigorous evidence chains, hypothesis comparison, and uncertainty handling. The authors propose JustDiag, a diagnostic argumentation engine that introduces, for the first time, an explicit modeling of the diagnostic reasoning process into root cause analysis. JustDiag structures and maintains states such as evidence, findings, competing hypotheses, conflicts, and follow-up checks to enable traceable and auditable inference, complemented by a calibration mechanism that explicitly accounts for uncertainty. Integrating LLMs with a structured reasoning framework, the approach employs a two-tier evaluation protocol to assess both outcome and reasoning quality. Experiments on 66 real-world incidents demonstrate that JustDiag significantly outperforms non-argumentative baselines in both outcome and process scores, exhibiting superior uncertainty retention despite a slightly lower completion rate.

accountabilitydiagnostic justificationincident response

This work addresses the challenges of root cause diagnosis in large-scale microservice systems, where existing approaches are hindered by massive log volumes, limited LLM context windows, and insufficient semantic reasoning and interpretability. The authors propose a neuro-symbolic hybrid method that emulates Site Reliability Engineers’ manual troubleshooting process through a six-stage pipeline for log sampling, template clustering, and anomaly ranking, producing a concise evidence package for LLM-based root cause inference. This approach compresses raw logs by 1,000–7,000× while preserving critical failure signals and provides auditable log templates and statistical evidence, substantially enhancing interpretability and practicality. Evaluated on 11 real-world incidents, the method achieves an MRR of 0.790 and ranks the correct root cause within the top three candidates in over 90% of cases within one minute, earning strong endorsement from operations teams.

incident diagnosislarge-scale systemslog analysis

This study addresses the challenges of root cause localization in complex cloud networks and the limited generalizability of traditional rule-based methods by proposing a novel paradigm that integrates spatiotemporal grouping, automated ontology construction, and a time-aware causal graph. The approach constructs the causal graph through bivariate Granger causality tests and conditional independence testing, and introduces an edge-specific time-lagged conditional probability inference mechanism to enable efficient and interpretable root cause scoring. Evaluated on 35 real-world production incidents, the method achieves a root cause recall rate of 85.7% and an exact match precision of 74.3%. It has been deployed in over 800 actual fault cases and received positive feedback from operations teams.

causal reasoningcloud networkslarge-scale systems

Hot Scholars

ZZ

Zibin Zheng

IEEE Fellow, Highly Cited Researcher, Sun Yat-sen University, China
BlockchainSmart ContractServices ComputingSoftware Reliability
XX

Xiaofei Xie

Singapore Management University
Software EngineeringLoop AnalysisTestingDeep Learning
MR

Michael R. Lyu

Professor of Computer Science & Engineering, The Chinese University of Hong Kong
software engineeringsoftware reliabilityfault tolerancemachine learning
TJ

Tong Jia

Peking University
AIOpsAnomaly DetectionLog AnalysisAI for Medical Research
NU

Nasir U. Eisty

Assistant Professor, University of Tennessee, Knoxville
Software EngineeringSoftware Quality AssuranceResearch Software EngineeringSoftware Security