fault tolerance and error handling

In cloud environments, selecting optimal data protection strategies for business continuity and disaster recovery remains challenging due to the lack of quantitative foundations for evaluating reliability and aligning with organizational Recovery Time Objectives (RTOs) and operational requirements. Method: This paper proposes an integrated assessment framework that synergistically combines system dynamics modeling and simulation-based optimization. It quantitatively evaluates key performance indicators—including recovery timeliness, data integrity, and system robustness—across public and hybrid cloud scenarios by simulating mainstream recovery mechanisms. Contribution/Results: The framework innovatively applies system dynamics to model time-varying dependencies during recovery processes and establishes interpretable, traceable mappings between policy parameters, technical metrics, and business objectives. Empirical validation demonstrates its reproducibility and practical utility, providing cloud-native organizations with a quantifiable, verifiable, and actionable decision-support methodology for data protection strategy selection.

To address insufficient resilience of complex systems under heterogeneous hardware environments, this paper proposes a fault-adaptive software deployment and redundancy configuration optimization method. We construct a system-level resilience state-space model and introduce a novel equivalence relation to enable quotient-space-based state-space reduction, significantly compressing the state space. Subsequently, we integrate formal model checking with strategy synthesis to automatically derive both an initial deployment configuration and dynamic reconfiguration policies that satisfy multi-level resilience requirements. Our key contributions are: (i) a new equivalence relation enabling efficient, semantics-preserving state-space reduction; and (ii) end-to-end automated synthesis of fault-response and recovery strategies. Experimental evaluation on an autonomous driving system model demonstrates that our approach substantially improves fault recovery latency and system availability, while supporting real-time resilience assurance.

Existing hybrid fault-tolerant systems exhibit limited flexibility in co-designing crash fault tolerance (CFT) and Byzantine fault tolerance (BFT), while incurring high redundancy overheads—e.g., redundant state replication and costly protocol switching. Method: This paper proposes ShellFT, the first framework enabling selective hybrid fault tolerance via micro-replication: it injects BFT protection only into critical components on-demand, while retaining lightweight CFT for all other modules—thereby decoupling fault models from system architecture. ShellFT supports dynamic BFT scope configuration through a customized protocol stack and fine-grained fault-domain partitioning. Contribution/Results: Experiments demonstrate that ShellFT reduces heterogeneous overheads by over 70% compared to conventional hybrid approaches, while sustaining high throughput and low latency. Its modular, adaptable design enables seamless deployment across diverse environments—including cloud-native platforms and edge computing infrastructures.

To address the challenges of standardizing Site Reliability Engineering (SRE) practices in heterogeneous environments and balancing system reliability with development agility, this paper proposes a customizable SRE process framework. The framework integrates automated operations, multidimensional observability (metrics, logs, traces), error-budget-driven governance, standardized incident response, and progressive delivery (canary and blue-green deployments). It is designed for cross-technology-stack adaptability, enabling contextual implementation of core SRE principles. Evaluated in production systems, the framework reduced mean time to recovery by 42%, decreased unplanned outages by 67%, lowered operational staffing requirements by 35%, and achieved 99.99% service availability. Its primary contribution is the first methodology for customizing SRE processes specifically for heterogeneous environments, empirically demonstrating synergistic improvements in both system reliability and operational efficiency.

Chaos engineering lacks a systematic, comprehensive review in the literature. Method: This paper conducts the first multi-source literature review (MLR), systematically analyzing 96 academic and gray literature sources published between 2016 and 2024—including 88 core publications from 2019 to 2024. It synthesizes findings via thematic clustering and qualitative coding. Contribution/Results: The study establishes the first consensus definition of chaos engineering, proposes a four-layer capability model and a five-dimensional component taxonomy, and performs a cross-tool evaluation of 12 mainstream chaos engineering tools. It identifies six open research challenges and clarifies practice drivers, tool characteristics, and research evolution trends. The results provide a foundational theoretical framework, methodological benchmark, and roadmap for future work—bridging critical knowledge gaps between academia and industry.

This study addresses the reliability challenges faced by modern web applications due to their inherent complexity and dynamic operating environments. The authors propose a modular self-healing framework grounded in the MAPE-K architecture, which innovatively integrates AutoFix-inspired heuristics with a learning-driven, feedback-guided recovery strategy to enable adaptive fault repair. Evaluated through fault injection experiments and iterative optimization in real-world scenarios, the system achieves an F1 score of 90.7% for fault detection and a 93.2% success rate in recovery, with an average recovery time of just 3.92 seconds. Notably, it sustains throughput at 88%–95% of baseline levels while increasing response time by only 3.1%, thereby significantly enhancing the resilience and autonomous recovery capabilities of web applications.

This study addresses the critical challenge that restoring IT backups alone is insufficient to resume production after ransomware attacks on manufacturing systems, due to deep interdependencies among IT, operational technology (OT), physical processes, identity management, and supply chains. The work reframes recovery as a problem of interdependent continuity in critical infrastructure and introduces, for the first time, the concept of “Minimum Viable Factory Recovery” (MVF Recovery), shifting the objective from full-system restoration to capability-oriented minimal trusted operations. Drawing on a PRISMA-guided multi-source systematic review integrating academic literature, standards, government guidelines, and real-world incidents, the study identifies nine failure modes in recovery efforts and proposes a capability-centered recovery framework. It further establishes an evidence-driven recovery lifecycle model and outlines directions for benchmarking, offering actionable recovery targets for critical manufacturing infrastructure.

This work addresses the challenge of cascading failures in modern microservice systems, where component restarts—often triggered by dense dependencies—can propagate faults, and existing autonomous repair mechanisms lack safety guarantees. The authors propose a novel three-agent collaborative architecture comprising diagnosis, planning, and verification agents that jointly generate repair plans encoded with explicit side-effect semantics via a typed set of seven atomic actions. These plans are executed and validated transactionally by a minimal kernel. Crucially, the approach introduces an online inference mechanism for recovery boundaries grounded in distributed tracing, which ensures operational safety while avoiding indiscriminate restarts. Empirical evaluation demonstrates that the system infers 99th-percentile recovery groups in just 21ms across Alibaba, Meta, and DeathStarBench datasets, reduces agent-induced harmful actions by 95% in simulations, and achieves zero harmful operations in production deployment.

Existing SRE benchmark tasks are overly simplified and fail to capture the complexity of fault diagnosis and mitigation in real-world production environments. This work proposes the first high-fidelity, scalable evaluation benchmark for SRE agents, built upon a realistic cloud-native system stack that dynamically simulates operational conditions. The benchmark incorporates a fault injector and noise simulator to support diverse failure modes—including metastable and correlated failures—and provides 90 realistic, challenging tasks. Designed with a modular architecture, it enables continuous extension and adaptation. Experimental results demonstrate significant performance disparities among state-of-the-art AI agents across different fault types, with end-to-end success rates varying by up to 40%, thereby validating the benchmark’s effectiveness and inherent difficulty.

This work addresses the challenge of meeting stringent Recovery Time Objective (RTO) requirements in distributed storage systems, where conventional content-hash-based synchronization mechanisms suffer from recovery delays due to hash index invalidation or reconstruction. To overcome this limitation, the paper proposes a lightweight, metadata-driven architecture that assigns globally unique, content-agnostic composite identifiers during write operations, thereby eliminating reliance on cryptographic hash computation. This approach enables efficient incremental synchronization without the need for hash recalculations, significantly reducing computational overhead during recovery and circumventing the hash-reconstruction bottleneck. Consequently, the proposed mechanism ensures deterministic and timely disaster recovery, effectively guaranteeing RTO compliance while enhancing the availability and resilience of distributed storage systems.