flink

Existing data stream processing frameworks face scalability bottlenecks on HPC infrastructures, and lack evaluation tools tailored for ultra-large-scale clusters. To address this, we propose the first modular, HPC-oriented stream processing benchmark suite. Our approach innovatively integrates native SLURM scheduling support, enabling seamless multi-framework compatibility with Apache Flink, Spark Streaming, and Kafka Streams. Leveraging a modular architecture and automated experiment orchestration, the suite supports end-to-end performance measurement and fully customizable configuration. Evaluated on real-world HPC clusters, our benchmark achieves over 10× higher throughput than state-of-the-art alternatives. It significantly improves the accuracy, reproducibility, and scalability of large-scale stream processing system evaluation. By bridging HPC and stream computing, this work establishes critical infrastructure for cross-disciplinary research in high-performance streaming systems.

In existing distributed stream processing frameworks, data processing logic is deeply coupled with distribution strategies, limiting scheduling flexibility in high-level frameworks and imposing excessive development overhead on low-level ones. Method: This paper proposes a pluggable stream processing framework that achieves full decoupling between processing logic and distribution strategies for the first time. It supports modular definition, reuse, and composition of distribution strategies via a domain-specific language (DSL). The framework integrates a runtime strategy plugin mechanism, streaming topology compilation optimizations, and a lightweight Storm-compatible execution engine. Contribution/Results: Experiments show that the framework achieves throughput comparable to Apache Storm, successfully reproduces multiple state-of-the-art distribution strategies, and validates the correctness, predictable performance, and engineering feasibility of modular strategy design—thereby bridging the critical gap between high-level abstraction expressiveness and low-level scheduling controllability.

This work addresses the challenges of slow failure recovery, poor stability, and high operational costs that Apache Flink faces in large-scale production environments, which hinder its ability to meet stringent service-level objectives (SLOs). To overcome these limitations, the authors propose the first systematic approach that integrates engine-level and cluster-level elasticity. The solution innovatively combines runtime optimizations, fine-grained fault tolerance, a hybrid replication strategy, and high-availability mechanisms leveraging external systems, complemented by a highly reliable automated testing and deployment pipeline. Evaluated on ByteDance’s ultra-large-scale Flink clusters, the proposed framework significantly enhances system elasticity, stability, and recovery efficiency, thereby effectively ensuring compliance with demanding SLOs.

Cloud application development has long suffered from high expertise barriers due to the need to integrate distributed systems, database, and software engineering knowledge. This paper proposes Stateflow—the first cloud-native, streaming, stateful function-computing framework—designed to jointly address programmability, strongly consistent fault-tolerant transactions, and serverless semantics. Its three key contributions are: (1) an object-oriented declarative programming model that eliminates explicit error handling; (2) the Styx engine, which guarantees deterministic multi-partition serializability, snapshot consistency, and zero-loss state migration; and (3) a streaming dataflow execution model with transaction-aware state migration, enabling dynamic elastic scaling. Experimental evaluation demonstrates that Stateflow significantly outperforms existing systems in both throughput and recovery performance, substantially reducing development complexity for high-concurrency, strongly consistent cloud applications.

In dynamic scientific workflows, the decoupling of task scheduling from data movement often assigns tasks to nodes lacking local input data, causing network congestion and execution delays. To address this, we propose the first workflow-aware joint scheduling framework that enables “data-readiness-driven” task placement via predictive pre-staging of intermediate data replicas, supporting dynamic execution plans. Our approach integrates speculative data pre-staging, dependency-aware scheduling, and lightweight storage management, implemented on a Nextflow+Kubernetes prototype. Experiments across 16 synthetic and real-world workflows demonstrate significant reductions in total completion time—up to 94.5% (synthetic) and 53.2% (real), with only bounded, transient storage overhead. The core contribution is the first holistic co-optimization of scheduling and data movement for dynamic workflows, breaking the traditional separation between these concerns.

To address poor scalability, high query latency, and tight architectural coupling in real-time spatial querying over massive mobile object streams, this paper proposes a streaming spatial query processing system built on Apache Flink Stateful Functions. Our approach introduces three key innovations: (1) a lightweight global grid index with efficient metadata synchronization, enabling low-overhead dynamic updates; (2) an actor-like stateful function model integrated with adaptive load balancing, achieving component decoupling and elastic horizontal scaling; and (3) a unified streaming spatial query processing paradigm supporting diverse query types—including range, k-nearest neighbor, and continuous trajectory queries—under a single framework. Experimental evaluation on both real-world and synthetic datasets demonstrates that the system sustains over one million object updates per second, delivers sub-100 ms query latency, and scales linearly to clusters of hundreds of nodes.

This work addresses the high network costs and operational overhead incurred by cross-availability-zone repartitioning in large-scale stream processing systems. It introduces, for the first time, cloud object storage into the stream shuffle pipeline through a pluggable architecture built atop Kafka Streams: upstream operators batch-write repartitioning data to object storage and emit lightweight notifications, while downstream operators fetch data on demand. By integrating configurable batching and distributed caching, the approach ensures consistency without requiring modifications to Kafka or underlying infrastructure. Experimental evaluation on AWS Kubernetes clusters demonstrates over 40× reduction in repartitioning costs, 95th-percentile latency under 2 seconds, and throughput exceeding 2 GiB/s, substantially enhancing scalability and efficiency.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

This work addresses key challenges in industrial IoT—namely, weak semantic interoperability, lack of context awareness, and rigid management mechanisms—when handling heterogeneous high-velocity data streams. The authors propose a context-aware knowledge graph platform that uniquely integrates knowledge graphs, contextual reasoning, and distributed stream processing based on Apache Kafka and Flink. This platform provides a unified semantic model encompassing devices, data streams, agents, processing pipelines, and permission roles, enabling flexible data ingestion, composable stream processing, and dynamic, context-driven access control. Experimental results demonstrate that the proposed approach significantly enhances system interoperability, adaptability, and context awareness, thereby establishing a semantic, interpretable, and secure data workflow infrastructure for Industry 5.0.

This work addresses the inefficiency of existing serverless computing and stream processing systems in handling short-lived, lightweight, and unpredictable stateful data streams. To bridge this gap, the paper proposes “stream functions”—an extension to the function-as-a-service model that elevates short streams to first-class units of execution, state management, and autoscaling. Stream functions express inter-event logic through iterator-based interfaces, effectively integrating stream processing semantics with the elasticity of serverless architectures. This approach is the first to treat short streams as fundamental execution units, thereby filling a critical void in lightweight stateful stream processing. Experimental evaluation in video processing scenarios demonstrates that the proposed system reduces runtime overhead by approximately 99% compared to mainstream stream processing engines, while maintaining performance comparable to conventional serverless functions.