incident response and management

In large-scale systems, on-call engineers rely on manual procedures or ad-hoc scripts for incident investigation, resulting in high mean time to resolution (MTTR), elevated operational overhead, and diminished productivity. This paper introduces DrP—the first end-to-end automated investigation framework designed for heterogeneous domains including services, AI/ML, and mobile systems. DrP’s key contributions are: (1) a declarative SDK enabling low-code development of reusable, domain-agnostic analysis logic; (2) a distributed execution engine with a plugin-based architecture supporting high-concurrency diagnostics and deep integration with alerting, event management, and remediation systems; and (3) a unified abstraction layer that transparently insulates users from infrastructure heterogeneity. Deployed at scale within Meta, DrP executes ~50,000 analyses daily across 300+ engineering teams, reducing average MTTR by 20% overall and up to 80% in specific scenarios—significantly enhancing SRE responsiveness and system observability.

Manual execution of Troubleshooting Guides (TSGs) in large-scale IT systems is inefficient and error-prone, while existing LLM-based approaches struggle with poor TSG quality, complex control flow, data-intensive queries, and parallel execution requirements. Method: We propose an end-to-end automation framework comprising: (i) TSG Mentor to enhance guide quality; (ii) an offline phase leveraging LLMs to construct a structured execution DAG and generate domain-specific Query Preparation Plugins (QPPs); and (iii) an online phase employing a DAG-guided, memory-augmented scheduler and executor that ensures correctness and enables task-level parallelism. Results: Evaluated on real-world TSGs and incidents, our framework achieves a 94% success rate with GPT-4.1—significantly outperforming baselines—and reduces execution time for parallelizable TSGs by 32.9%–70.4%, while also improving token efficiency and latency.

Current AI agent incident reporting mechanisms suffer from critical limitations: they rely solely on publicly available data, omitting sensitive yet essential internal execution traces—such as reasoning chains and tool invocation logs—thereby hindering identification of root causes. This work pioneers the application of systems safety principles to AI agent incident analysis, proposing a novel “Systemic–Contextual–Cognitive” tri-dimensional causal framework. We design an integrated attribution methodology combining activity log analysis, system documentation review, and tool behavior tracing. Furthermore, we specify mandatory fields for incident reports and define a minimal sensitive dataset that developers and deployers must retain—including reasoning trajectories, API call sequences, and environmental context. Our contributions establish both a theoretical foundation and practical guidelines for building explainable, reproducible, and intervenable AI agent incident response mechanisms. (149 words)

This work addresses the challenges of root cause diagnosis in large-scale microservice systems, where existing approaches are hindered by massive log volumes, limited LLM context windows, and insufficient semantic reasoning and interpretability. The authors propose a neuro-symbolic hybrid method that emulates Site Reliability Engineers’ manual troubleshooting process through a six-stage pipeline for log sampling, template clustering, and anomaly ranking, producing a concise evidence package for LLM-based root cause inference. This approach compresses raw logs by 1,000–7,000× while preserving critical failure signals and provides auditable log templates and statistical evidence, substantially enhancing interpretability and practicality. Evaluated on 11 real-world incidents, the method achieves an MRR of 0.790 and ranks the correct root cause within the top three candidates in over 90% of cases within one minute, earning strong endorsement from operations teams.

Existing event management approaches for large-scale microservice systems rely on unimodal data, struggle to jointly perform anomaly detection, failure classification, and root-cause localization, and lack interpretability. To address these limitations, this paper proposes the first end-to-end multi-task joint optimization framework for automated fault management. Our method introduces a novel three-channel heterogeneous data processing pipeline that fuses multimodal telemetry—metrics, logs, and distributed traces—and incorporates an LLM-augmented collaborative reasoning mechanism to enable modality-specific feature extraction and unified training. Evaluated on two mainstream microservice benchmarks, our framework achieves improvements of 4.7%–57.7% in anomaly detection, 2.1%–40.6% in failure attribution, and 1.6%–163.1% in root-cause localization. Moreover, it generates verifiable, step-by-step reasoning evidence, substantially enhancing model interpretability and operational utility.

This work addresses the limited interpretability and accountability of large language models (LLMs) in root cause analysis, which hinder their applicability in high-stakes operational settings requiring rigorous evidence chains, hypothesis comparison, and uncertainty handling. The authors propose JustDiag, a diagnostic argumentation engine that introduces, for the first time, an explicit modeling of the diagnostic reasoning process into root cause analysis. JustDiag structures and maintains states such as evidence, findings, competing hypotheses, conflicts, and follow-up checks to enable traceable and auditable inference, complemented by a calibration mechanism that explicitly accounts for uncertainty. Integrating LLMs with a structured reasoning framework, the approach employs a two-tier evaluation protocol to assess both outcome and reasoning quality. Experiments on 66 real-world incidents demonstrate that JustDiag significantly outperforms non-argumentative baselines in both outcome and process scores, exhibiting superior uncertainty retention despite a slightly lower completion rate.

This work addresses a critical gap in microservice fault diagnosis: while existing methods can accurately identify root causes, they often fail to generate effective and executable recovery actions, preventing true system restoration. To bridge this gap, the authors propose R2Act, a novel framework that formally defines a recovery-oriented action space, introduces metrics for action effectiveness, and establishes an offline evaluation protocol. They also construct a benchmark dataset comprising 302 real-world Kubernetes faults, annotated with root causes and synchronized multimodal observations. Leveraging techniques such as action modeling and retrieval-augmented generation (RAG) enhanced large language models (LLMs), the study systematically evaluates the entire pipeline from diagnosis to recovery. Experimental results reveal that despite root cause localization accuracy ranging from 91.4% to 99.7%, the effectiveness of generated recovery actions remains limited at only 36.8%–60.3%, highlighting a key bottleneck in current LLM-based recovery decision-making.

This work addresses the vulnerability of existing LLM-driven microservice root cause analysis (RCA) agents to early reasoning errors that lead to diagnostic failure and their inability to localize or correct such mistakes. The study introduces a novel formulation of RCA errors as stage-localizable reasoning flaws and proposes a structured four-stage framework—comprising evidence bundles, hypothesis sets, analytical structures, and decision reports. To enhance robustness, it integrates stage-level auditing, budget-aware fast-slow path routing, counterfactual candidate evaluation, and stage-specific patch replay mechanisms. Implemented via LangGraph, the system demonstrates significant improvements in root cause localization and fault classification accuracy on both public benchmarks and real-world production data, precisely identifying erroneous stages and successfully repairing most execution trajectories within one to two iterations, thereby substantially improving agent debuggability and self-repair capability.

This work addresses a critical gap in the safety mechanisms of large language model (LLM) agents, which predominantly focus on preventive measures and lack capabilities for responding to and recovering from incidents after they occur. To bridge this gap, the authors propose AIR—the first incident response framework tailored for LLM agents—that integrates semantic anomaly detection, tool-driven containment, and recovery mechanisms within the agent’s execution loop. AIR also leverages a domain-specific language to automatically generate protective rules, enabling autonomous management across the full incident lifecycle. Experimental evaluation demonstrates that AIR achieves over 90% success rates in detection, remediation, and eradication across three representative agent types, with automatically generated rules performing comparably to handcrafted ones, all while maintaining acceptable runtime overhead.

This work addresses the significant challenge of real-time identification of actionable technical risks from high-noise, high-throughput, and semantically complex customer event streams in enterprise systems. The authors propose an end-to-end risk discovery system that innovatively integrates large language models (LLMs) with an efficient indexing mechanism. Through a multi-stage event linking strategy, cascaded routing, and a multidimensional denoising pipeline—combining domain knowledge, statistical patterns, and behavioral filtering—the system achieves precise, low-latency risk extraction and accurate business attribution. Evaluated in a real-world production environment processing an average of 300,000 events per day with peaks of 2,000 events per minute, the system demonstrates a P90 alert latency of only 3.5 minutes and a 95% detection rate for high-priority events, substantially outperforming existing baseline approaches.