π€ AI Summary
This work addresses the high network costs and operational overhead incurred by cross-availability-zone repartitioning in large-scale stream processing systems. It introduces, for the first time, cloud object storage into the stream shuffle pipeline through a pluggable architecture built atop Kafka Streams: upstream operators batch-write repartitioning data to object storage and emit lightweight notifications, while downstream operators fetch data on demand. By integrating configurable batching and distributed caching, the approach ensures consistency without requiring modifications to Kafka or underlying infrastructure. Experimental evaluation on AWS Kubernetes clusters demonstrates over 40Γ reduction in repartitioning costs, 95th-percentile latency under 2 seconds, and throughput exceeding 2 GiB/s, substantially enhancing scalability and efficiency.
π Abstract
Shuffling or repartitioning data streams is an essential operation of state-of-the-art stream processing frameworks to support stateful workloads in a large-scale, distributed setting. In today's cloud deployments, however, shuffling can become a major cost driver due to substantial network traffic across multiple availability zones (AZs) as well as an operational burden when operating a high-throughput, strongly consistent messaging backbone at scale. We present BlobShuffle, a novel approach to cost-effective shuffling for stream processing systems that leverages cloud object storage as an intermediate exchange layer. Instead of sending all shuffled records directly, BlobShuffle groups records into batches, stores these batches in cloud object storage, and forwards only compact notifications. Downstream operators use these notifications to retrieve the relevant batches and extract the corresponding records. BlobShuffle balances cost efficiency and latency through configurable batching and a distributed caching mechanism. BlobShuffle is implemented as an add-on for Kafka Streams that requires only minimal code changes to existing applications, leaves Kafka and the underlying infrastructure unmodified, and preserves Kafka Streams' consistency and correctness guarantees. In a large-scale experimental evaluation on a Kubernetes-based AWS deployment, we show that BlobShuffle can reduce shuffling costs by more than 40x compared to native Kafka Streams shuffling while keeping the 95th percentile shuffle latency below 2 seconds. Moreover, it scales to processing more than 2 GiB/s without encountering a scalability limit in our experiments, indicating that BlobShuffle can economically support shuffle-intensive workloads at large scale.