HEAL: Online Incremental Recovery for Leaderless Distributed Systems Across Persistency Models

2026-02-09

AI Summary

This work addresses the high recovery overhead and latency in leaderless, non-transactional distributed systems during node failures, which severely degrade online throughput. To this end, the authors propose HEAL—the first generic, lightweight, online incremental recovery mechanism tailored for such systems. By integrating linearizability with diverse memory persistence strategies and employing an optimized incremental recovery algorithm, HEAL significantly reduces recovery costs while preserving consistency guarantees. Experimental evaluation on an Intel cluster demonstrates that HEAL achieves an average recovery time of only 120 milliseconds, with a mere 8.7% reduction in runtime throughput. Compared to conventional and leader-based approaches, HEAL accelerates recovery by up to 3,000× and nearly halves the throughput degradation.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Multiagent Systems: Distributed Problem Solving

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Ensuring resilience in distributed systems has become an acute concern. In today's environment, it is crucial to develop light-weight mechanisms that recover a distributed system from faults quickly and with only a small impact on the live-system throughput. To address this need, this paper proposes a new low-overhead, general recovery scheme for modern non-transactional leaderless distributed systems. We call our scheme HEAL. On a node failure, HEAL performs an optimized online incremental recovery. This paper presents HEAL's algorithms for settings with Linearizable consistency and different memory persistency models. We implement HEAL on a 6-node Intel cluster. Our experiments running TAOBench workloads show that HEAL is very effective. HEAL recovers the cluster in 120 milliseconds on average, while reducing the throughput of the running workload by an average of 8.7%. In contrast, a conventional recovery scheme for leaderless systems needs 360 seconds to recover, reducing the throughput of the system by 16.2%. Finally, compared to an incremental recovery scheme for a state-of-the-art leader-based system, HEAL reduces the average recovery latency by 20.7x and the throughput degradation by 62.4%.

Problem

• distributed systems
• fault recovery
• leaderless
• incremental recovery
• persistency models

Innovation

• incremental recovery
• leaderless distributed systems
• linearizable consistency
• memory persistency models
• fault tolerance