ConflictSync: Bandwidth Efficient Synchronization of Divergent State

2025-05-02

AI Summary

To address the high communication overhead and unbounded metadata growth inherent in full-state synchronization of state-based CRDTs, this paper introduces the first digest-driven synchronization paradigm. It models synchronization as a redundancy-free, decomposable set reconciliation problem, generalizing rate-distortion set reconciliation to variable-length elements for the first time. By integrating Bloom filter pre-filtering with invertible Bloom lookup tables (IBLTs), it overcomes efficiency bottlenecks in low-similarity scenarios. Compared to conventional state synchronization, total transmission volume is reduced by up to 18×; Bloom pre-filtering cuts communication cost by 50% even at 0% similarity; and an optimal trade-off between similarity and Bloom filter size is analytically derived. The approach eliminates the need for external metadata garbage collection and maintains efficient incremental synchronization following network partitions.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Applications of cryptography
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

State-based Conflict-free Replicated Data Types (CRDTs) are widely used in distributed systems to ensure high availability without coordination. However, their naive synchronization strategy - transmitting the full state - incurs high communication costs. Existing optimizations like delta-CRDTs reduce this overhead but rely on external metadata that must be garbage collected to prevent unbounded growth, at the cost of full state transmissions after network partitions. This paper presents ConflictSync, the first digest-driven synchronization algorithm for state-based CRDTs. We reduce synchronization to the set reconciliation of irredundant join decompositions and build on existing work in rateless set reconciliation. To support CRDTs, we generalize set reconciliation to variable-sized elements, and further introduce a novel combination of Bloom filters with Rateless Invertible Bloom Lookup Tables to address inefficiencies at low similarity levels. Our evaluation shows that ConflictSync reduces total data transfer by up to 18 times compared to traditional state-based synchronization. Bloom filter prefiltering reduces overhead by up to 50% compared to pure rateless reconciliation at 0% similarity, while pure rateless reconciliation performs better above 93% similarity. We characterize the trade-off between similarity level and Bloom filter size, identifying optimal configurations for different synchronization scenarios. Although developed for CRDTs, ConflictSync applies to any synchronization problem where states can be decomposed into sets of constituent components, analogous to join decompositions, making it suitable for a wide range of distributed data models.

Problem

• Reduces high communication costs in state-based CRDT synchronization
• Eliminates need for garbage collection of external metadata
• Optimizes data transfer for varying similarity levels

Innovation

• Digest-driven synchronization for state-based CRDTs
• Generalized set reconciliation for variable-sized elements
• Bloom filters with Rateless Invertible Bloom Lookup Tables