Skitter: A Distributed Stream Processing Framework with Pluggable Distribution Strategies

2025-02-15

The Art, Science, and Engineering of Programming

AI Summary

In existing distributed stream processing frameworks, data processing logic is deeply coupled with distribution strategies, limiting scheduling flexibility in high-level frameworks and imposing excessive development overhead on low-level ones. Method: This paper proposes a pluggable stream processing framework that achieves full decoupling between processing logic and distribution strategies for the first time. It supports modular definition, reuse, and composition of distribution strategies via a domain-specific language (DSL). The framework integrates a runtime strategy plugin mechanism, streaming topology compilation optimizations, and a lightweight Storm-compatible execution engine. Contribution/Results: Experiments show that the framework achieves throughput comparable to Apache Storm, successfully reproduces multiple state-of-the-art distribution strategies, and validates the correctness, predictable performance, and engineering feasibility of modular strategy design—thereby bridging the critical gap between high-level abstraction expressiveness and low-level scheduling controllability.

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Context: Distributed Stream Processing Frameworks (DSPFs) are popular tools for expressing real-time Big Data applications that have to handle enormous volumes of data in real time. These frameworks distribute their applications over a cluster in order to scale horizontally along with the amount of incoming data. Inquiry: Crucial for the performance of such applications is the **distribution strategy** that is used to partition data and computations over the cluster nodes. In some DSPFs, like Apache Spark or Flink, the distribution strategy is hardwired into the framework which can lead to inefficient applications. The other end of the spectrum is offered by Apache Storm, which offers a low-level model wherein programmers can implement their own distribution strategies on a per-application basis to improve efficiency. However, this model conflates distribution and data processing logic, making it difficult to modify either. As a consequence, today's cluster application developers either have to accept the built-in distribution strategies of a high-level framework or accept the complexity of expressing a distribution strategy in Storm's low-level model. Approach: We propose a novel programming model wherein data processing operations and their distribution strategies are decoupled from one another and where new strategies can be created in a modular fashion. Knowledge: The introduced language abstractions cleanly separate the data processing and distribution logic of a stream processing application. This enables the expression of stream processing applications in a high-level framework while still retaining the flexibility offered by Storm's low-level model. Grounding: We implement our programming model as a domain-specific language, called Skitter, and use it to evaluate our approach. Our evaluation shows that Skitter enables the implementation of existing distribution strategies from the state of the art in a modular fashion. Our performance evaluation shows that the strategies implemented in Skitter exhibit the expected performance characteristics and that applications written in Skitter obtain throughput rates in the same order of magnitude as Storm. Importance: Our work enables developers to select the most performant distribution strategy for each operation in their application, while still retaining the programming model offered by high-level frameworks.

Problem

• Decouples data processing and distribution strategies in DSPFs.
• Enables modular creation of new distribution strategies.
• Improves flexibility and performance in stream processing applications.

Innovation

• Decouples data processing and distribution strategies
• Introduces modular, pluggable distribution strategies
• Enables high-level programming with low-level flexibility