apache kafka

The lack of systematic integration between Kafka design patterns and benchmarking methodologies hinders reproducible, evidence-based architectural decision-making for event-streaming systems. Method: This study systematically analyzes 42 academic and industrial publications (2015–2025) using bibliometric analysis and pattern induction, complemented by standardized (TPCx-Kafka, Yahoo Streaming Benchmark) and customized workload evaluations. Contribution/Results: We introduce the first unified Kafka architectural pattern taxonomy—comprising nine high-frequency patterns—and establish a pattern-to-benchmark mapping matrix. Additionally, we propose heuristic guidelines to support architecture-level decisions. Our work fills a critical gap in reproducible design methodology for event-streaming systems, significantly improving design quality and practical consistency across performance, fault tolerance, and cross-study comparability dimensions.

This work addresses the practical challenges of deploying machine learning models in real-world settings, where heterogeneous data protocols, non-standard formats, and infrastructure constraints often necessitate redundant construction of integration pipelines. To overcome these issues, we propose SMOCS—a containerized, streaming ML system built on Apache Kafka—that decouples infrastructure from application logic through layered abstraction and employs a three-threaded agent architecture to separate data ingestion, online training, and real-time inference. The framework enables configuration-driven, no-code deployment, offering platform independence, fault isolation, and horizontal scalability, thereby significantly lowering the barrier to entry for domain experts. SMOCS has been open-sourced on the Jefferson Lab GitHub repository and demonstrates both continuous online learning capability and strong engineering practicality.

Existing data stream processing frameworks face scalability bottlenecks on HPC infrastructures, and lack evaluation tools tailored for ultra-large-scale clusters. To address this, we propose the first modular, HPC-oriented stream processing benchmark suite. Our approach innovatively integrates native SLURM scheduling support, enabling seamless multi-framework compatibility with Apache Flink, Spark Streaming, and Kafka Streams. Leveraging a modular architecture and automated experiment orchestration, the suite supports end-to-end performance measurement and fully customizable configuration. Evaluated on real-world HPC clusters, our benchmark achieves over 10× higher throughput than state-of-the-art alternatives. It significantly improves the accuracy, reproducibility, and scalability of large-scale stream processing system evaluation. By bridging HPC and stream computing, this work establishes critical infrastructure for cross-disciplinary research in high-performance streaming systems.

This work addresses the non-determinism in real-time stream processing—caused by factors such as scheduling, window triggering, out-of-order data, and network jitter—which hinders auditability and reproducibility. The authors propose a lightweight blockchain-based provenance architecture that, for the first time, integrates blockchain as a cryptographic anchor into Kafka Streams. Instead of storing raw payloads on-chain, the system records only Merkle roots of windowed data along with offset metadata, preserving data privacy and system performance while enabling verifiable integrity, sequence consistency, and analytical correctness of processing results. Experiments using a real-world weather dataset from Berlin demonstrate that the approach supports fully deterministic replay, incurs linearly scalable verification overhead, and achieves high-throughput blockchain integration, effectively meeting the demands of real-time stream processing scenarios.

To address insufficient multi-objective load balancing in stream processing systems under complex workloads, this paper proposes a multi-tier collaborative scheduling framework. The framework introduces dynamic inter-layer coordination mechanisms and lightweight interfaces among schedulers, enabling seamless integration of novel scheduling policies. It jointly optimizes computational resource utilization, end-to-end latency, and throughput by integrating multi-objective optimization, distributed resource management, and real-time feedback control. Its key innovation lies in shifting hierarchical scheduling from static decoupling to dynamic collaboration—preserving scalability while significantly enhancing adaptability. Evaluated in Meta’s production environment, the system reliably processes TB-scale data with sub-second latency; it improves critical resource utilization by 27% and reduces tail latency by 41%.

Existing large model training data pipelines struggle to simultaneously ensure batch semantics, fault isolation, and consistency. This work proposes an agent-free, object-storage-native training data plane featuring three core innovations: a Transactional Global Batch (TGB) abstraction that guarantees training consistency, a storage-layer-embedded garbage collection mechanism aligning producer states with distributed checkpoints, and a communication-free Decentralized Adaptive Commit (DAC) algorithm. Leveraging lakehouse ACID semantics in object storage and distributed checkpointing, the system achieves higher throughput than colocated loaders and Kafka, lower read latency, and full fault isolation across 64-GPU multimodal pretraining and supervised fine-tuning (SFT) workloads.

This work addresses the limitations of traditional real-time analytics systems, which rely on manually defined queries and struggle to proactively uncover the vast array of potential insights within complex, dynamic data streams. To overcome this, the authors propose a multi-agent architecture that establishes a continuous closed-loop process for autonomous insight discovery, encompassing hypothesis generation, compilation of executable analyses, result validation, and visualization. A key innovation is the introduction of a contract-driven design based on typed intermediate artifacts, which ensures modularity, observability, lineage tracking, and secure execution of dynamic analyses. The system leverages Kafka as its event coordination backbone and Flink for stream processing, integrating large language models to power specialized agents. Empirical evaluations in retail, financial, and public data scenarios demonstrate an effective paradigm shift from query-driven to proactive discovery-driven analytics.

Existing RDF systems lack standardized message boundaries in event-driven and streaming contexts, resulting in poor interoperability and inefficient processing. This work proposes the concept of “RDF Messages,” formally defining, for the first time, a message structure with RDF datasets as atomic units and explicitly delineating message boundaries to support streaming, storage, and reproducible replay. Built upon W3C Semantic Web standards, the approach introduces a unified encapsulation mechanism that operates across serialization and transport layers, complemented by configuration profiles that describe message structure and processing policies. The proposed framework effectively enables incremental consumption in scenarios such as IoT observations and SPARQL CONSTRUCT results, laying the foundation for RDF message streams and logs while advancing related W3C standardization efforts.

This work addresses the high network costs and operational overhead incurred by cross-availability-zone repartitioning in large-scale stream processing systems. It introduces, for the first time, cloud object storage into the stream shuffle pipeline through a pluggable architecture built atop Kafka Streams: upstream operators batch-write repartitioning data to object storage and emit lightweight notifications, while downstream operators fetch data on demand. By integrating configurable batching and distributed caching, the approach ensures consistency without requiring modifications to Kafka or underlying infrastructure. Experimental evaluation on AWS Kubernetes clusters demonstrates over 40× reduction in repartitioning costs, 95th-percentile latency under 2 seconds, and throughput exceeding 2 GiB/s, substantially enhancing scalability and efficiency.

This work proposes a serverless real-time data processing framework that integrates the MapReduce programming model with Function-as-a-Service (FaaS) to address the demand for efficient handling of massive, real-time data streams in modern logistics systems. Built on Kubernetes and Knative, the framework employs an event-driven architecture composed of five loosely coupled services for data ingestion, aggregation, and analysis. It leverages Apache Kafka for event transport, Redis for metadata management, and AWS S3 for durable storage. By innovatively combining MapReduce’s batch-processing semantics with FaaS’s elastic scaling capabilities, the system supports on-demand autoscaling and scale-to-zero functionality. Experimental results demonstrate that the proposed approach achieves low-latency processing while significantly improving resource efficiency, thereby fulfilling the requirements of highly elastic and highly available data pipelines.