CertainSync: Rateless Set Reconciliation with Certainty

2025-04-11

AI Summary

In distributed systems such as blockchain, transaction pool synchronization requires estimating the size of the symmetric difference between sets, leading to uncontrollable coordination failure rates. Method: This paper proposes the first parameter-free, deterministic set reconciliation framework that eliminates the need for symmetric difference estimation. It integrates invertible Bloom lookup tables (IBLTs), information-theoretic lower-bound-driven adaptive transmission, and UniverseReduceSync—a universe-reduction hashing scheme—achieving both determinism and communication efficiency in large-scale settings. Contribution/Results: We prove 100% reconciliation success for multiple IBLT constructions. Simulations demonstrate that communication overhead approaches the information-theoretic lower bound. Empirical evaluation on real Ethereum transaction hash data confirms substantial reduction in communication cost compared to state-of-the-art approaches.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Multiagent Systems: Mechanism Design

Application Category

• Security and Privacy: Blockchains and distributed ledgers
• Economics, Online Markets and Human Computation: Economic aspects of blockchain and cryptocurrencies
• Web Mining and Content Analysis: Content-based information diffusion

Abstract

Set reconciliation is a fundamental task in distributed systems, particularly in blockchain networks, where it enables synchronization of transaction pools among peers and facilitates block dissemination. Traditional set reconciliation schemes are either statistical, offering success probability as a function of communication overhead and symmetric difference size, or require parametrization and estimation of that size, which can be error-prone. We present CertainSync, a novel reconciliation framework that, to the best of our knowledge, is the first to guarantee successful set reconciliation without any parametrization or estimators. The framework is rateless and adapts to the unknown symmetric difference size. Reconciliation is guaranteed whenever the communication overhead reaches a lower bound derived from the symmetric difference size and universe size. Our framework builds on recent constructions of Invertible Bloom Lookup Tables (IBLTs), ensuring successful element listing as long as the number of elements is bounded. We provide a theoretical analysis proving the certainty of reconciliation for multiple constructions. Our approach is validated by simulations, showing the ability to synchronize sets with efficient communication costs while maintaining guarantees compared to baseline schemes. To further reduce overhead in large universes such as blockchain networks, CertainSync is extended with a universe reduction technique. We compare and validate this extension, UniverseReduceSync, against the basic framework using real Ethereum transaction hash data. Results show a trade-off between lower communication costs and maintaining guarantees, offering a comprehensive solution for diverse reconciliation scenarios.

Problem

• Guarantees successful set reconciliation without parametrization
• Adapts to unknown symmetric difference size efficiently
• Reduces communication overhead in large blockchain networks

Innovation

• Rateless set reconciliation without parametrization
• Adapts to unknown symmetric difference size
• Uses Invertible Bloom Lookup Tables (IBLTs)