SProBench: Stream Processing Benchmark for High Performance Computing Infrastructure

2025-04-03

AI Summary

Existing data stream processing frameworks face scalability bottlenecks on HPC infrastructures, and lack evaluation tools tailored for ultra-large-scale clusters. To address this, we propose the first modular, HPC-oriented stream processing benchmark suite. Our approach innovatively integrates native SLURM scheduling support, enabling seamless multi-framework compatibility with Apache Flink, Spark Streaming, and Kafka Streams. Leveraging a modular architecture and automated experiment orchestration, the suite supports end-to-end performance measurement and fully customizable configuration. Evaluated on real-world HPC clusters, our benchmark achieves over 10× higher throughput than state-of-the-art alternatives. It significantly improves the accuracy, reproducibility, and scalability of large-scale stream processing system evaluation. By bridging HPC and stream computing, this work establishes critical infrastructure for cross-disciplinary research in high-performance streaming systems.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Time-Series/Data Streams
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Recent advancements in data stream processing frameworks have improved real-time data handling, however, scalability remains a significant challenge affecting throughput and latency. While studies have explored this issue on local machines and cloud clusters, research on modern high performance computing (HPC) infrastructures is yet limited due to the lack of scalable measurement tools. This work presents SProBench, a novel benchmark suite designed to evaluate the performance of data stream processing frameworks in large-scale computing systems. Building on best practices, SProBench incorporates a modular architecture, offers native support for SLURM-based clusters, and seamlessly integrates with popular stream processing frameworks such as Apache Flink, Apache Spark Streaming, and Apache Kafka Streams. Experiments conducted on HPC clusters demonstrate its exceptional scalability, delivering throughput that surpasses existing benchmarks by more than tenfold. The distinctive features of SProBench, including complete customization options, built-in automated experiment management tools, seamless interoperability, and an open-source license, distinguish it as an innovative benchmark suite tailored to meet the needs of modern data stream processing frameworks.

Problem

• Evaluating scalability of stream processing in HPC systems
• Addressing lack of tools for large-scale performance measurement
• Benchmarking throughput and latency in modern frameworks

Innovation

• Modular architecture for scalable stream processing
• Native SLURM support for HPC clusters
• Seamless integration with Apache frameworks