🤖 AI Summary
This work addresses the non-determinism in real-time stream processing—caused by factors such as scheduling, window triggering, out-of-order data, and network jitter—which hinders auditability and reproducibility. The authors propose a lightweight blockchain-based provenance architecture that, for the first time, integrates blockchain as a cryptographic anchor into Kafka Streams. Instead of storing raw payloads on-chain, the system records only Merkle roots of windowed data along with offset metadata, preserving data privacy and system performance while enabling verifiable integrity, sequence consistency, and analytical correctness of processing results. Experiments using a real-world weather dataset from Berlin demonstrate that the approach supports fully deterministic replay, incurs linearly scalable verification overhead, and achieves high-throughput blockchain integration, effectively meeting the demands of real-time stream processing scenarios.
📝 Abstract
Auditability and reproducibility still are critical challenges for real-time data streams pipelines. Streaming engines are highly dependent on runtime scheduling, window triggers, arrival orders, and uncertainties such as network jitters. These all derive the streaming pipeline platforms to throw non-determinist outputs. In this work, we introduce a blockchain-backed provenance architecture for streaming platform (e.g Kafka Streams) the publishes cryptographic data of a windowed data stream without publishing window payloads on-chain. We used real-time weather data from weather stations in Berlin. Weather records are canonicalized, deduplicated, and aggregated per window, then serialised deterministically. Furthermore, the Merkle root of the records within the window is computed and stored alongside with Kafka offsets boundaries to MultiChain blockchain streams as checkpoints. Our design can enable an independent auditor to verify: (1) the completeness of window payloads, (2) canonical serialization, and (3) correctness of derived analytics such as minimum/maximum/average temperatures. We evaluated our system using real data stream from two weather stations (Berlin-Brandenburg and Berlin-Tempelhof) and showed linear verification cost, deterministic reproducibility, and with a scalable off-chain storage with on-chain cryptographic anchoring. We also demonstrated that the blockchain can afford to be integrated with streaming platforms particularly with our system, and we get satisfactory transactions per second values.