reliability engineering

To address the challenges of standardizing Site Reliability Engineering (SRE) practices in heterogeneous environments and balancing system reliability with development agility, this paper proposes a customizable SRE process framework. The framework integrates automated operations, multidimensional observability (metrics, logs, traces), error-budget-driven governance, standardized incident response, and progressive delivery (canary and blue-green deployments). It is designed for cross-technology-stack adaptability, enabling contextual implementation of core SRE principles. Evaluated in production systems, the framework reduced mean time to recovery by 42%, decreased unplanned outages by 67%, lowered operational staffing requirements by 35%, and achieved 99.99% service availability. Its primary contribution is the first methodology for customizing SRE processes specifically for heterogeneous environments, empirically demonstrating synergistic improvements in both system reliability and operational efficiency.

To address challenges in cloud-native communication/networking services—including ambiguous SLI/SLO definitions, high expertise barriers for specialized monitoring, and low cross-organizational trust in metrics—this paper proposes the first SRE platform integrating generative AI, federated learning, and blockchain. Methodologically, it introduces federated learning for collaborative, privacy-preserving SLI metric discovery across distributed environments; employs QLoRA-finetuned Llama-3-8B to enable intelligent, context-aware SLI/SLO generation; and leverages smart contracts and NFTs on-chain to immutably attest and audit metrics. The platform is compatible with Prometheus/Mimir, supports lightweight deployment, and was validated on Open5GS 5G core network, demonstrating effective automated SLO management. It simultaneously ensures data privacy, system transparency, and engineering practicality.

SLA monitoring in service-oriented architectures faces a trust dilemma: self-reported compliance by service providers risks undetected violations. This paper proposes the first verifiable SLA monitoring framework leveraging trusted execution environments (TEEs) and zero-knowledge proofs (ZKPs), enabling trustless verification of violation claims. Our contributions are threefold: (1) a semantic translation mechanism that converts machine-readable SLA clauses into formally verifiable assertions; (2) a privacy-preserving verification protocol integrating timestamped telemetry, Merkle-tree aggregation, and digital signatures—ensuring compliance validation without exposing raw data; and (3) ZKP generation and verification optimized to near-constant time complexity. The prototype achieves over one million monitored events per hour, demonstrating high scalability alongside strong security guarantees. The framework ensures integrity, authenticity, and validity of SLA enforcement, thereby strengthening accountability in distributed service ecosystems.

This work addresses the challenge of modeling and governing user journey reliability in microservice systems, an emergent property that existing SLO-as-code approaches fail to capture effectively, leading to misalignment between intended objectives and actual system behavior. To bridge this gap, the paper introduces Emergence-as-Code (EmaC), a novel framework that formalizes journey reliability as declarative, verifiable, and executable code specifications. By integrating distributed tracing, traffic analysis, and runtime model synthesis, EmaC automatically infers journey-level SLOs and their error budgets, then leverages an SLO compiler to generate corresponding control-plane artifacts. Embedded within GitOps workflows, EmaC enables auditable, autonomous governance mechanisms—such as burn rate alerts and release gates—to achieve computable, closed-loop management of end-to-end reliability.

Existing SRE benchmark tasks are overly simplified and fail to capture the complexity of fault diagnosis and mitigation in real-world production environments. This work proposes the first high-fidelity, scalable evaluation benchmark for SRE agents, built upon a realistic cloud-native system stack that dynamically simulates operational conditions. The benchmark incorporates a fault injector and noise simulator to support diverse failure modes—including metastable and correlated failures—and provides 90 realistic, challenging tasks. Designed with a modular architecture, it enables continuous extension and adaptation. Experimental results demonstrate significant performance disparities among state-of-the-art AI agents across different fault types, with end-to-end success rates varying by up to 40%, thereby validating the benchmark’s effectiveness and inherent difficulty.

Manual execution of Troubleshooting Guides (TSGs) in large-scale IT systems is inefficient and error-prone, while existing LLM-based approaches struggle with poor TSG quality, complex control flow, data-intensive queries, and parallel execution requirements. Method: We propose an end-to-end automation framework comprising: (i) TSG Mentor to enhance guide quality; (ii) an offline phase leveraging LLMs to construct a structured execution DAG and generate domain-specific Query Preparation Plugins (QPPs); and (iii) an online phase employing a DAG-guided, memory-augmented scheduler and executor that ensures correctness and enables task-level parallelism. Results: Evaluated on real-world TSGs and incidents, our framework achieves a 94% success rate with GPT-4.1—significantly outperforming baselines—and reduces execution time for parallelizable TSGs by 32.9%–70.4%, while also improving token efficiency and latency.

Existing MVCC serializability protocols (e.g., SSN) rely on a single serialization point—typically commit time—and thus fail to accommodate read-write anti-dependencies inherent in Snapshot Isolation (SI), leading to excessive transaction aborts. Method: We propose a relaxed commit-time validation mechanism that, for the first time, incorporates total transaction ordering into multiversion serializability checking. Our approach jointly leverages the Direct Serialization Graph (DSG) and Multiversion Serialization Graph (MVSG) criteria under invariant semantics, enabling a single lightweight check. It preserves version-chain monotonicity, eliminating costly chain traversal while maintaining storage overhead comparable to SSN. Contribution/Results: Our method strictly subsumes SSN’s correctness guarantees and supports snapshot reads ordered by either start or commit time. Experiments on reproducible mixed workloads show an absolute reduction of 0.25 and a relative decrease of ~50% in long-transaction abort rates, significantly improving concurrency performance.

This work addresses the heavy reliance on manual intervention in operating large-scale Elasticsearch clusters by introducing ES Guardian Agent, an autonomous运维 framework that integrates system metrics, application logs, and kernel-level telemetry (e.g., dmesg, NVMe SMART) into a multi-source fault prediction engine. Coupled with a tool-augmented large language model, it implements a five-layer monitoring architecture and an iterative AI action loop to achieve end-to-end lifecycle autonomy—including deployment, tuning, monitoring, diagnosis, repair, and upgrades. Evaluated in production, the system autonomously executed 300 repairs, recovered an 18-hour cross-system outage, detected a cluster-wide NIC failure, and identified per-shard data volume as the dominant factor in query latency (0.26 ms increase per MB/shard), thereby supporting a six-nines availability target.

This work addresses the vulnerability of the real-time operating system FreeRTOS to ionizing radiation-induced faults in safety-critical applications, which poses a significant threat to system reliability. The authors propose KRONOS, a non-intrusive software-based fault injection framework that requires neither specialized hardware nor debugging interfaces, enabling the first systematic assessment of the susceptibility of FreeRTOS kernel data structures—such as scheduler variables and Task Control Blocks (TCBs)—to both transient and permanent faults. Through large-scale, targeted fault injection experiments, the study reveals that corruption of pointers and critical scheduler variables frequently leads to system crashes, whereas damage to most TCB fields has limited impact on system availability. These findings provide crucial insights for designing more resilient and reliable real-time operating systems.

This work addresses the lack of closed-loop control in traditional software development lifecycles, which often fails to simultaneously ensure security, auditability, and highly reliable automation. The authors propose a deterministic autonomous control framework that models the lifecycle as a seven-stage automated pipeline, integrating Jira-based task orchestration, structured context, resource constraints, and human-review gating mechanisms to establish a secure closed loop. Key innovations include a state-contract-based collision locking mechanism, a degradation protocol for fallback operation, and a traceable control architecture. Implemented with 12,661 lines of Python code and 6,907 lines of versioned prompt specifications—including 101 exception handlers and 12 centralized locks—the system achieved a 100% success rate (95% CI [97.6%, 100%]) across 152 initial runs, producing over 795 artifacts. All 51 issues identified through adversarial review were fully resolved, with 60% of security tickets autonomously completed.