stream processing

Stream data processing faces persistent challenges including manual incremental maintenance, SQL semantic inconsistencies between streaming and batch paradigms, and insufficient enterprise-grade operational capabilities; moreover, existing systems overemphasize sub-second latency, limiting applicability to mainstream second- to minute-scale analytical workloads. To address these, we propose Delayed View Semantics (DVS), the first formal framework unifying stream and batch semantics across a broad latency spectrum. We design declarative dynamic table primitives to ensure end-to-end transactional consistency and high availability, and extend the snapshot isolation model to enforce invariants in streaming applications. Evaluated on a production system built atop Snowflake, our approach reduces development complexity by over 80%, supports configurable end-to-end latency from milliseconds to minutes, cuts operational overhead by 90%, and currently serves thousands of enterprise customers.

In existing distributed stream processing frameworks, data processing logic is deeply coupled with distribution strategies, limiting scheduling flexibility in high-level frameworks and imposing excessive development overhead on low-level ones. Method: This paper proposes a pluggable stream processing framework that achieves full decoupling between processing logic and distribution strategies for the first time. It supports modular definition, reuse, and composition of distribution strategies via a domain-specific language (DSL). The framework integrates a runtime strategy plugin mechanism, streaming topology compilation optimizations, and a lightweight Storm-compatible execution engine. Contribution/Results: Experiments show that the framework achieves throughput comparable to Apache Storm, successfully reproduces multiple state-of-the-art distribution strategies, and validates the correctness, predictable performance, and engineering feasibility of modular strategy design—thereby bridging the critical gap between high-level abstraction expressiveness and low-level scheduling controllability.

Existing large model training data pipelines struggle to simultaneously ensure batch semantics, fault isolation, and consistency. This work proposes an agent-free, object-storage-native training data plane featuring three core innovations: a Transactional Global Batch (TGB) abstraction that guarantees training consistency, a storage-layer-embedded garbage collection mechanism aligning producer states with distributed checkpoints, and a communication-free Decentralized Adaptive Commit (DAC) algorithm. Leveraging lakehouse ACID semantics in object storage and distributed checkpointing, the system achieves higher throughput than colocated loaders and Kafka, lower read latency, and full fault isolation across 64-GPU multimodal pretraining and supervised fine-tuning (SFT) workloads.

Existing data stream processing frameworks face scalability bottlenecks on HPC infrastructures, and lack evaluation tools tailored for ultra-large-scale clusters. To address this, we propose the first modular, HPC-oriented stream processing benchmark suite. Our approach innovatively integrates native SLURM scheduling support, enabling seamless multi-framework compatibility with Apache Flink, Spark Streaming, and Kafka Streams. Leveraging a modular architecture and automated experiment orchestration, the suite supports end-to-end performance measurement and fully customizable configuration. Evaluated on real-world HPC clusters, our benchmark achieves over 10× higher throughput than state-of-the-art alternatives. It significantly improves the accuracy, reproducibility, and scalability of large-scale stream processing system evaluation. By bridging HPC and stream computing, this work establishes critical infrastructure for cross-disciplinary research in high-performance streaming systems.

Existing LLM frameworks are stateless and execute queries in isolation, making them ill-suited for long-horizon, semantics-aware analysis of unstructured data streams. Method: This paper introduces the first LLM-driven continuous stream processing paradigm, extending Retrieval-Augmented Generation (RAG) to streaming settings and formalizing *continuous semantic operators*. We design a dynamic optimization framework integrating lightweight shadow execution with multi-objective Bayesian optimization (MOBO) to adaptively balance throughput and accuracy. Further, we incorporate tuple batching, operator fusion, and cost-aware scheduling within the VectraFlow system. Results: Experiments demonstrate that VectraFlow responds to load fluctuations in real time, sustains high-accuracy continuous semantic querying under high throughput, achieves significant efficiency gains, and incurs only bounded, controllable accuracy degradation.

This work addresses the non-determinism in real-time stream processing—caused by factors such as scheduling, window triggering, out-of-order data, and network jitter—which hinders auditability and reproducibility. The authors propose a lightweight blockchain-based provenance architecture that, for the first time, integrates blockchain as a cryptographic anchor into Kafka Streams. Instead of storing raw payloads on-chain, the system records only Merkle roots of windowed data along with offset metadata, preserving data privacy and system performance while enabling verifiable integrity, sequence consistency, and analytical correctness of processing results. Experiments using a real-world weather dataset from Berlin demonstrate that the approach supports fully deterministic replay, incurs linearly scalable verification overhead, and achieves high-throughput blockchain integration, effectively meeting the demands of real-time stream processing scenarios.

This work addresses the inefficiency of existing serverless computing and stream processing systems in handling short-lived, lightweight, and unpredictable stateful data streams. To bridge this gap, the paper proposes “stream functions”—an extension to the function-as-a-service model that elevates short streams to first-class units of execution, state management, and autoscaling. Stream functions express inter-event logic through iterator-based interfaces, effectively integrating stream processing semantics with the elasticity of serverless architectures. This approach is the first to treat short streams as fundamental execution units, thereby filling a critical void in lightweight stateful stream processing. Experimental evaluation in video processing scenarios demonstrates that the proposed system reduces runtime overhead by approximately 99% compared to mainstream stream processing engines, while maintaining performance comparable to conventional serverless functions.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.

This work addresses the challenges of slow failure recovery, poor stability, and high operational costs that Apache Flink faces in large-scale production environments, which hinder its ability to meet stringent service-level objectives (SLOs). To overcome these limitations, the authors propose the first systematic approach that integrates engine-level and cluster-level elasticity. The solution innovatively combines runtime optimizations, fine-grained fault tolerance, a hybrid replication strategy, and high-availability mechanisms leveraging external systems, complemented by a highly reliable automated testing and deployment pipeline. Evaluated on ByteDance’s ultra-large-scale Flink clusters, the proposed framework significantly enhances system elasticity, stability, and recovery efficiency, thereby effectively ensuring compliance with demanding SLOs.

This work addresses the high latency in real-time stream processing caused by tight coupling between state I/O and the data path, which blocks the CPU on the critical path. To mitigate this, the authors propose Keyed Prefetching, a mechanism that extracts state-access keys from upstream operators and proactively prefetches the required state, thereby overlapping I/O with computation to hide latency. Complementing this, a Timestamp-Aware Caching strategy is introduced to efficiently manage prefetched and historical states in memory. Together, these techniques significantly reduce end-to-end latency for long-running real-time queries while maintaining high throughput and effectively decoupling state access from data processing.