Holon Streaming: Global Aggregations with Windowed CRDTs
🤖 AI Summary
To address scalability limitations, single-point bottlenecks, high latency, and slow failure recovery caused by centralized coordination in exactly-once stream processing for global aggregations, this paper proposes a decentralized stream processing architecture based on Windowed Conflict-Free Replicated Data Types (CRDTs). The architecture ensures strong consistency, state convergence, and fault-tolerant robustness for global aggregations via distributed state replication, a deterministic computation model, and decentralized failure recovery. Its key innovation is the first application of Windowed CRDTs to stream processing, enabling coordination-free incremental aggregation and cross-node state synchronization. Experimental results demonstrate that, under global aggregation workloads, the system achieves a 2× throughput improvement and a 5× reduction in end-to-end latency compared to baseline systems. During failure recovery, latency is further reduced by 11×, significantly outperforming existing exactly-once stream processors.
Application Category
📝 Abstract
Scaling global aggregations is a challenge for exactly-once stream processing systems. Current systems implement these either by computing the aggregation in a single task instance, or by static aggregation trees, which limits scalability and may become a bottleneck. Moreover, the end-to-end latency is determined by the slowest path in the tree, and failures and reconfiguration cause large latency spikes due to the centralized coordination. Towards these issues, we present Holon Streaming, an exactly-once stream processing system for global aggregations. Its deterministic programming model uses windowed conflict-free replicated data types (Windowed CRDTs), a novel abstraction for shared replicated state. Windowed CRDTs make computing global aggregations scalable. Furthermore, their guarantees such as determinism and convergence enable the design of efficient failure recovery algorithms by decentralized coordination. Our evaluation shows a 5x lower latency and 2x higher throughput than an existing stream processing system on global aggregation workloads, with an 11x latency reduction under failure scenarios. The paper demonstrates the effectiveness of decentralized coordination with determinism, and the utility of Windowed CRDTs for global aggregations.
Problem
Research questions and friction points this paper is trying to address.
Scaling global aggregations in exactly-once stream processing systems
Overcoming bottlenecks of static aggregation trees and centralized coordination
Reducing latency spikes caused by failures and system reconfiguration
Innovation
Methods, ideas, or system contributions that make the work stand out.
Windowed CRDTs enable scalable global aggregations
Decentralized coordination reduces latency and improves throughput
Deterministic programming model ensures exactly-once stream processing
🔎 Similar Papers
2024-05-28Citations: 0
