Score
Applying a hypothesis‑driven approach to diagnose and resolve engineering issues by collecting reproducible evidence (logs, metrics, stack traces), isolating root causes, iterating fixes or mitigations, and documenting resolutions to prevent recurrence.
A significant gap exists between academic research and industrial practice in debugging machine learning (ML) systems. Method: We propose the first comprehensive, lifecycle-spanning taxonomy of ML debugging faults and corresponding mitigation methods, derived from a systematic literature review (SLR), in-depth interviews with 28 ML practitioners, and empirical analysis of 1,247 GitHub issues. Contribution/Results: Our study identifies 13 core debugging challenges; only 48% are addressed by existing academic work, while 52.6% of GitHub issues and 70.3% of interview-elicited problems lack corresponding methodological support. Critically, we quantitatively demonstrate that over half of real-world ML debugging difficulties remain unaddressed by current research—revealing a substantial knowledge gap. This work establishes a foundational classification framework, provides empirical evidence of methodological coverage gaps, and delivers a prioritized roadmap to bridge the theory-practice divide in ML debugging.
This work addresses the limited interpretability and accountability of large language models (LLMs) in root cause analysis, which hinder their applicability in high-stakes operational settings requiring rigorous evidence chains, hypothesis comparison, and uncertainty handling. The authors propose JustDiag, a diagnostic argumentation engine that introduces, for the first time, an explicit modeling of the diagnostic reasoning process into root cause analysis. JustDiag structures and maintains states such as evidence, findings, competing hypotheses, conflicts, and follow-up checks to enable traceable and auditable inference, complemented by a calibration mechanism that explicitly accounts for uncertainty. Integrating LLMs with a structured reasoning framework, the approach employs a two-tier evaluation protocol to assess both outcome and reasoning quality. Experiments on 66 real-world incidents demonstrate that JustDiag significantly outperforms non-argumentative baselines in both outcome and process scores, exhibiting superior uncertainty retention despite a slightly lower completion rate.
This work addresses the lack of automated, structured failure-recovery mechanisms in current software engineering agents, which struggle to translate heterogeneous runtime evidence into actionable repair guidance. The paper proposes PROBE, a novel framework that introduces a failure-anchored, structured recovery paradigm. PROBE employs a three-layer architecture—telemetry, diagnosis, and guidance gate—to decouple yet coordinate diagnosis and recovery, enabling non-intrusive integration. By integrating runtime telemetry, multi-signal diagnosis, and evidence-driven bounded guidance generation, PROBE constructs an end-to-end recovery pipeline. Evaluated on 257 unresolved cases, PROBE achieves a Top-1 diagnostic accuracy of 65.37% and a recovery success rate of 21.79%, significantly outperforming the strongest baseline. Its practical feasibility has been validated through deployment in Microsoft’s IcM system.
Root cause analysis (RCA) for cross-modal telemetry (metrics, logs, traces) in microservice systems suffers from slow diagnosis, poor interpretability, and lack of actionable remediation guidance. To address these challenges, we propose the first multimodal RCA framework integrating statistical causal discovery with large language model (LLM) agents. Our method constructs a causal-graph-guided iterative reasoning workflow that bridges suspicious service ranking to causally interpretable and operationally actionable repair recommendations; it further introduces graph-augmented LLM inference and joint modeling of heterogeneous telemetry sources. Evaluated on an open-source benchmark, our approach improves root cause localization accuracy by 42.22%. Human evaluation confirms that its diagnostic outputs exhibit significantly higher causal plausibility and operational feasibility, effectively narrowing the semantic gap between automated diagnosis and frontline incident response.
CI/CD pipeline failure diagnosis and repair have long suffered from high complexity and low automation. This paper introduces LogSage—the first end-to-end, LLM-driven framework for root-cause analysis (RCA) and automated repair. It features a novel two-stage LLM architecture: (1) Stage I employs intelligent log preprocessing to precisely localize failures; (2) Stage II integrates retrieval-augmented generation (RAG) with tool calling to generate executable, validated fixes. LogSage is the first industrial-grade solution validated on over one million production CI/CD pipelines. It achieves 98% RCA accuracy—12 percentage points higher than state-of-the-art baselines—and end-to-end repair accuracy exceeding 88%. Deployed at scale, it supported 1.07 million CI/CD executions in its first year, processing over 3,000 tasks daily.
This work proposes an intelligent agent-based diagnostic framework leveraging large language models (LLMs) to overcome the limitations of traditional root cause analysis methods, which rely on hard-coded rules, incur high maintenance costs, and are tightly coupled with infrastructure. By integrating a Model Context Protocol (MCP) and a constrained tool space, the framework enables agents to autonomously invoke tools for service querying, dependency retrieval, and multi-source data analysis, facilitating stepwise reasoning to pinpoint root causes. A structured investigation protocol ensures traceable and reproducible inference while maintaining robustness under incomplete or ambiguous information, effectively decoupling the model from underlying infrastructure. This approach lays the foundation for autonomous fault diagnosis and change impact assessment, paving the way for automated remediation and risk prediction, thereby significantly enhancing operational efficiency and system safety.
This work addresses the pervasive issue of redundant and isolated messages in system logs, which hinder downstream tasks such as model reasoning and anomaly detection. To tackle this challenge, the authors propose LogPurifier—the first task-agnostic log cleansing framework—that systematically purifies logs by extracting log templates and modeling their dependencies to accurately identify and remove messages irrelevant to system functional behavior. By doing so, LogPurifier enables effective log sanitization applicable across diverse analytical scenarios. Experimental results demonstrate that LogPurifier substantially improves both accuracy and efficiency in various downstream tasks, thereby validating its effectiveness and generalizability.
This work addresses a critical gap in microservice fault diagnosis: while existing methods can accurately identify root causes, they often fail to generate effective and executable recovery actions, preventing true system restoration. To bridge this gap, the authors propose R2Act, a novel framework that formally defines a recovery-oriented action space, introduces metrics for action effectiveness, and establishes an offline evaluation protocol. They also construct a benchmark dataset comprising 302 real-world Kubernetes faults, annotated with root causes and synchronized multimodal observations. Leveraging techniques such as action modeling and retrieval-augmented generation (RAG) enhanced large language models (LLMs), the study systematically evaluates the entire pipeline from diagnosis to recovery. Experimental results reveal that despite root cause localization accuracy ranging from 91.4% to 99.7%, the effectiveness of generated recovery actions remains limited at only 36.8%–60.3%, highlighting a key bottleneck in current LLM-based recovery decision-making.
Existing approaches to root cause analysis in Kubernetes often rely on scenario-specific heuristics, lacking auditability and reproducibility. This work proposes a graph-guided root cause analysis agent that systematically encodes operational constraints—such as read-only forensics, propagation-aware diagnosis, and independent validation—into a typed evidence graph and a LangGraph state machine. By integrating large language model reasoning with deterministic graph traversal over OpenTelemetry telemetry data, the method enables an auditable and reproducible diagnostic workflow. Evaluated on 23 scenarios from the ITBench benchmark, the approach improves the F1 score for root cause entities from 0.6087 to 0.9130; even without scenario-specific prompts, it maintains a score of 0.6958, demonstrating both effectiveness and partial generalization capability.
This work addresses the challenges of root cause diagnosis in large-scale microservice systems, where existing approaches are hindered by massive log volumes, limited LLM context windows, and insufficient semantic reasoning and interpretability. The authors propose a neuro-symbolic hybrid method that emulates Site Reliability Engineers’ manual troubleshooting process through a six-stage pipeline for log sampling, template clustering, and anomaly ranking, producing a concise evidence package for LLM-based root cause inference. This approach compresses raw logs by 1,000–7,000× while preserving critical failure signals and provides auditable log templates and statistical evidence, substantially enhancing interpretability and practicality. Evaluated on 11 real-world incidents, the method achieves an MRR of 0.790 and ranks the correct root cause within the top three candidates in over 90% of cases within one minute, earning strong endorsement from operations teams.
System-level software packages frequently fail to build due to evolving toolchains and architectural diversity, with repairs hindered by multilingual components, dependency constraints, and architecture-specific nuances. This work presents the first systematic analysis of real-world build failures, revealing that 72% stem from dependency and environment misconfigurations. To address this, we propose an evidence-preserving iterative repair framework that decouples evidence management from tool execution, integrating external reproducible build services, modular fault localization, a repair controller, and a knowledge-driven architecture adaptation mechanism. Evaluated on RISC-V, aarch64, and x86_64 platforms, our approach achieves repair success rates of 53.88%, 41.77%, and 46.99%, respectively—substantially outperforming agent-based baselines (20.55%) and direct LLM approaches (1.83%).