data processing frameworks

In existing distributed stream processing frameworks, data processing logic is deeply coupled with distribution strategies, limiting scheduling flexibility in high-level frameworks and imposing excessive development overhead on low-level ones. Method: This paper proposes a pluggable stream processing framework that achieves full decoupling between processing logic and distribution strategies for the first time. It supports modular definition, reuse, and composition of distribution strategies via a domain-specific language (DSL). The framework integrates a runtime strategy plugin mechanism, streaming topology compilation optimizations, and a lightweight Storm-compatible execution engine. Contribution/Results: Experiments show that the framework achieves throughput comparable to Apache Storm, successfully reproduces multiple state-of-the-art distribution strategies, and validates the correctness, predictable performance, and engineering feasibility of modular strategy design—thereby bridging the critical gap between high-level abstraction expressiveness and low-level scheduling controllability.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.

Existing data stream processing frameworks face scalability bottlenecks on HPC infrastructures, and lack evaluation tools tailored for ultra-large-scale clusters. To address this, we propose the first modular, HPC-oriented stream processing benchmark suite. Our approach innovatively integrates native SLURM scheduling support, enabling seamless multi-framework compatibility with Apache Flink, Spark Streaming, and Kafka Streams. Leveraging a modular architecture and automated experiment orchestration, the suite supports end-to-end performance measurement and fully customizable configuration. Evaluated on real-world HPC clusters, our benchmark achieves over 10× higher throughput than state-of-the-art alternatives. It significantly improves the accuracy, reproducibility, and scalability of large-scale stream processing system evaluation. By bridging HPC and stream computing, this work establishes critical infrastructure for cross-disciplinary research in high-performance streaming systems.

This work addresses the performance overhead incurred by intermediate object creation in consecutive map() and filter() operations of the Java Stream API. The authors propose a compile-time optimization technique that transforms such operation chains into a single mapMulti() invocation via bytecode rewriting, thereby eliminating intermediate objects. This approach overcomes limitations of existing tools in handling lambda escape and stream variable assignments, and notably does not rely on loop unrolling. Experimental evaluation shows that the method significantly outperforms Streamliner on 2 out of 9 benchmarks while matching its performance on the rest. Furthermore, it successfully passes all 31,799 unit tests in Apache Kafka, demonstrating both correctness and practical applicability.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.

This work addresses the lack of precise semantic definitions and static error checking in the Vega visualization language, which often leads users to produce erroneous or ambiguous visualizations. For the first time, the paper formalizes Vega’s reactive dataflow mechanism by introducing a graph-based operational semantics and designing a complementary type system that precisely captures its streaming data transformation behavior. This approach enables static type analysis capable of preventing a wide range of common errors. The authors implement a type checker and formally verify its soundness with respect to the proposed semantics, thereby establishing the first formal framework for declarative visualization languages that combines rigorous semantics with strong static guarantees.

The era of large language models (LLMs) faces critical challenges including insufficient high-quality data supply, fragmented data preparation pipelines, poor reproducibility, and lack of model-in-the-loop support. Method: We propose the first LLM-driven, unified data preparation framework for data-centric AI, featuring system-level abstractions and PyTorch-style APIs for modular design. We introduce DataFlow-Agent—the first agent that synthesizes executable data pipelines end-to-end from natural language specifications—and integrate LLM-powered operator synthesis, iterative validation, 200+ reusable operators, and six domain-agnostic pipeline templates. Results: Experiments on Text-to-SQL, code generation, and mathematical reasoning show our synthesized data significantly outperforms human-annotated and domain-specific synthetic data. Remarkably, just 10K samples surpass the performance of models trained on the million-scale Infinity-Instruct dataset, empirically validating the decisive impact of data quality on model performance.

Cloud application development has long suffered from high expertise barriers due to the need to integrate distributed systems, database, and software engineering knowledge. This paper proposes Stateflow—the first cloud-native, streaming, stateful function-computing framework—designed to jointly address programmability, strongly consistent fault-tolerant transactions, and serverless semantics. Its three key contributions are: (1) an object-oriented declarative programming model that eliminates explicit error handling; (2) the Styx engine, which guarantees deterministic multi-partition serializability, snapshot consistency, and zero-loss state migration; and (3) a streaming dataflow execution model with transaction-aware state migration, enabling dynamic elastic scaling. Experimental evaluation demonstrates that Stateflow significantly outperforms existing systems in both throughput and recovery performance, substantially reducing development complexity for high-concurrency, strongly consistent cloud applications.

Existing large model training data pipelines struggle to simultaneously ensure batch semantics, fault isolation, and consistency. This work proposes an agent-free, object-storage-native training data plane featuring three core innovations: a Transactional Global Batch (TGB) abstraction that guarantees training consistency, a storage-layer-embedded garbage collection mechanism aligning producer states with distributed checkpoints, and a communication-free Decentralized Adaptive Commit (DAC) algorithm. Leveraging lakehouse ACID semantics in object storage and distributed checkpointing, the system achieves higher throughput than colocated loaders and Kafka, lower read latency, and full fault isolation across 64-GPU multimodal pretraining and supervised fine-tuning (SFT) workloads.