Distributed Download from an External Data Source in Faulty Majority Settings

2024-12-27

AI Summary

This paper addresses the problem of efficiently downloading complete information from a trusted external data source in Byzantine fault-tolerant distributed networks: under a fully connected topology with up to βk malicious nodes, it ensures that all honest nodes eventually obtain the full dataset while minimizing their maximum query count and communication time. The authors propose the first protocol achieving optimal communication round complexity under a dynamic adversary model. They design two complementary algorithms—randomized (for Byzantine failures) and deterministic (for crash failures)—that jointly leverage broadcast and point-to-point communication, augmented with failure-model-adaptive mechanisms. Theoretical analysis demonstrates near-optimal query complexity of O((n/k)·log n), optimal or asymptotically optimal time complexity, and significantly reduced load on honest nodes across diverse synchrony assumptions (synchronous, partially synchronous, asynchronous) and failure models (static, dynamic).

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We extend the study of retrieval problems in distributed networks, focusing on improving the efficiency and resilience of protocols in the emph{Data Retrieval (DR) Model}. The DR Model consists of a complete network (i.e., a clique) with $k$ peers, up to $eta k$ of which may be Byzantine (for $eta in [0, 1)$), and a trusted emph{External Data Source} comprising an array $X$ of $n$ bits ($n gg k$) that the peers can query. Additionally, the peers can also send messages to each other. In this work, we focus on the Download problem that requires all peers to learn $X$. Our primary goal is to minimize the maximum number of queries made by any honest peer and additionally optimize time. We begin with a randomized algorithm for the Download problem that achieves optimal query complexity up to a logarithmic factor. For the stronger dynamic adversary that can change the set of Byzantine peers from one round to the next, we achieve the optimal time complexity in peer-to-peer communication but with larger messages. In broadcast communication where all peers (including Byzantine peers) are required to send the same message to all peers, with larger messages, we achieve almost optimal time and query complexities for a dynamic adversary. Finally, in a more relaxed crash fault model, where peers stop responding after crashing, we address the Download problem in both synchronous and asynchronous settings. Using a deterministic protocol, we obtain nearly optimal results for both query complexity and message sizes in these scenarios.

Problem

• Byzantine Faults
• Efficient Data Retrieval
• Network Stability

Innovation

• Distributed Download Strategy
• Byzantine Fault Tolerance
• Optimal Communication Efficiency