operations

This work addresses the lack of systematic evaluation of large language models (LLMs) in AIOps scenarios. We introduce AIOpsBench—the first multilingual, multitask benchmark for IT operations—covering three core tasks: fault root-cause analysis, operations script generation, and alert summarization, with 7,184 multiple-choice and 1,736 open-ended questions. Methodologically, we propose the first systematic taxonomy of Ops capabilities; enforce rigorous expert validation and strict test-set isolation to ensure reliability; and integrate hallucination detection, automated QA evaluation, and a dynamic leaderboard. Key contributions include: empirical characterization of how model scale, quantization, and training strategies affect operational competence; open-sourcing 20% high-quality annotated data; and releasing a real-time, multilingual performance leaderboard. All benchmark data and evaluation framework are publicly available.

To address the challenges of standardizing Site Reliability Engineering (SRE) practices in heterogeneous environments and balancing system reliability with development agility, this paper proposes a customizable SRE process framework. The framework integrates automated operations, multidimensional observability (metrics, logs, traces), error-budget-driven governance, standardized incident response, and progressive delivery (canary and blue-green deployments). It is designed for cross-technology-stack adaptability, enabling contextual implementation of core SRE principles. Evaluated in production systems, the framework reduced mean time to recovery by 42%, decreased unplanned outages by 67%, lowered operational staffing requirements by 35%, and achieved 99.99% service availability. Its primary contribution is the first methodology for customizing SRE processes specifically for heterogeneous environments, empirically demonstrating synergistic improvements in both system reliability and operational efficiency.

Operational inefficiencies arise from inconsistent language, disorganized formatting, and execution deviations in Standard Operating Procedure (SOP) documents, while traditional modeling approaches impose excessive technical barriers for non-expert users. Method: This paper proposes SOPStruct—a novel framework that enables end-to-end automatic transformation of unstructured SOPs into decision-tree–structured representations using large language models (LLMs). It introduces a dual-track evaluation system integrating PDDL-based formal verification and LLM-driven semantic assessment to jointly ensure structural correctness and semantic completeness. Contribution/Results: SOPStruct supports standardized, cross-domain modeling of SOPs with varying complexity. Experiments demonstrate significant reductions in user cognitive load, improved accuracy in process comprehension, enhanced execution reliability, and effective support for automated workflow orchestration and human-auditable error correction.

In large-scale systems, on-call engineers rely on manual procedures or ad-hoc scripts for incident investigation, resulting in high mean time to resolution (MTTR), elevated operational overhead, and diminished productivity. This paper introduces DrP—the first end-to-end automated investigation framework designed for heterogeneous domains including services, AI/ML, and mobile systems. DrP’s key contributions are: (1) a declarative SDK enabling low-code development of reusable, domain-agnostic analysis logic; (2) a distributed execution engine with a plugin-based architecture supporting high-concurrency diagnostics and deep integration with alerting, event management, and remediation systems; and (3) a unified abstraction layer that transparently insulates users from infrastructure heterogeneity. Deployed at scale within Meta, DrP executes ~50,000 analyses daily across 300+ engineering teams, reducing average MTTR by 20% overall and up to 80% in specific scenarios—significantly enhancing SRE responsiveness and system observability.

Manual execution of Troubleshooting Guides (TSGs) in large-scale IT systems is inefficient and error-prone, while existing LLM-based approaches struggle with poor TSG quality, complex control flow, data-intensive queries, and parallel execution requirements. Method: We propose an end-to-end automation framework comprising: (i) TSG Mentor to enhance guide quality; (ii) an offline phase leveraging LLMs to construct a structured execution DAG and generate domain-specific Query Preparation Plugins (QPPs); and (iii) an online phase employing a DAG-guided, memory-augmented scheduler and executor that ensures correctness and enables task-level parallelism. Results: Evaluated on real-world TSGs and incidents, our framework achieves a 94% success rate with GPT-4.1—significantly outperforming baselines—and reduces execution time for parallelizable TSGs by 32.9%–70.4%, while also improving token efficiency and latency.

This work addresses the safety and reliability challenges of autonomous decision-making by large language models in NetOps/AIOps by proposing a constrained autonomy–centric agent-based operations framework. The framework defines clear boundaries for agent observation, proposal, and execution through enforceable contracts and integrates evidence collection, policy adherence, access control, and rollback mechanisms to establish a closed-loop workflow spanning diagnosis, root cause analysis, configuration generation, and limited self-healing. Full-process safety and controllability are achieved via sandbox replay, canary testing, constrained tool invocation, and auditable traceability. By establishing design principles for reliable, auditable, and secure deployment of intelligent operations, this study advances evaluation criteria from static question-answering toward end-to-end robustness and safety verification.

This work addresses the limitations of existing large language models (LLMs) in supporting efficient end-to-end intelligent operations due to low-quality data and fragmented domain knowledge. To overcome these challenges, the authors propose OpsLLM, a domain-specific LLM tailored for software operations, which introduces an innovative human-in-the-loop data construction pipeline and a Domain Process Reward Model (DPRM). The model is optimized through supervised fine-tuning combined with reinforcement learning. The project releases open-source models at multiple scales alongside a high-quality dataset, establishing a new paradigm for building end-to-end operational LLMs. Experimental results demonstrate that OpsLLM achieves accuracy improvements of 0.2%–5.7% on question-answering tasks and 2.7%–70.3% on root cause analysis tasks, significantly outperforming both open-source and closed-source baselines while exhibiting strong generalization capabilities.

This study addresses critical challenges in Security Operations Centers, including overwhelming threat alerts, heterogeneous SIEM platforms, and inefficient manual analysis. To overcome these issues, the authors propose SQM, an end-to-end large language model framework that innovatively integrates syntax-constrained prompting, ensemble-based threat detection, and retrieval-augmented generation to automatically produce executable queries and high-accuracy response recommendations. Experimental results demonstrate that SQM achieves a threat detection accuracy of 82.8% with a false positive rate of 0.120 across mainstream SIEM platforms. The framework attains a BLEU score of 0.384 and ROUGE-L of 0.731 for query generation, while boosting response recommendation accuracy to 90.0% and reducing average analyst triage time from several hours to under 10 minutes.

This work addresses the fragmentation in existing frameworks that treat deterministic and probabilistic computations in isolation, lacking a unified declarative language to orchestrate large language models (LLMs) and symbolic tools. We propose Structured Prompt Language (SPL), the first framework to deeply integrate probabilistic operations (GENERATE/EVALUATE) and deterministic reasoning (SOLVE/ASSERT) within a single declarative paradigm. SPL supports shared variable binding, runtime dynamic routing, and seamless interoperability with LLMs (e.g., Ollama, Anthropic), symbolic engines (e.g., SymPy, SageMath, Lean), and the distributed execution grid Momagrid. Across 1,200 experiments, SPL achieves machine-verified correctness rates of 82–93% (e.g., 93% for gemma4:e2b), substantially outperforming pure LLM baselines; most failures stem from solver kernels rejecting invalid expressions.

This work addresses the challenge in customer service automation where coupling routine requests with complex operations often leads to execution errors. The authors propose a difficulty-aware routing architecture that employs a lightweight classifier to direct simple interactions along an efficient baseline path, while routing high-risk, multi-entity, or backend-conflicting requests to an enhanced workflow. This enriched path triggers conflict-aware dialogues and a re-evaluation mechanism prior to critical write operations. By applying stringent controls only when necessary, the approach avoids global overhead while substantially improving system reliability. Evaluated on the τ²-bench suite across retail and airline domains, the architecture effectively ensures accurate record binding, supports fallback strategies, and orchestrates multi-step operations in a robust and orderly manner.