LOG.io: Unified Rollback Recovery and Data Lineage Capture for Distributed Data Pipelines

2025-12-17

AI Summary

Existing serverless distributed data pipelines struggle to simultaneously achieve efficient rollback recovery and fine-grained data lineage capture. Method: This paper proposes a log-driven, non-blocking, integrated mechanism that unifies lightweight event-level lineage tracking with operator-level asynchronous rollback. It supports non-deterministic operations, external system interactions, and dynamic scaling—while enabling failure recovery without interrupting ongoing operator execution. The design leverages structured logging, asynchronous non-blocking protocols, and runtime dynamic scheduling. Contribution/Results: Our approach eliminates the global blocking overhead inherent in traditional barrier-based snapshotting. Experiments under medium throughput (100 ms/event) and long-tail operator workloads show superior recovery performance compared to Flink ABS; lineage instrumentation overhead remains consistently below 1.5%; and data parallelism effectively amortizes processing overhead.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Planning, Routing, and Scheduling: Scheduling under Uncertainty

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper introduces LOG.io, a comprehensive solution designed for correct rollback recovery and fine-grain data lineage capture in distributed data pipelines. It is tailored for serverless scalable architectures and uses a log-based rollback recovery protocol. LOG.io supports a general programming model, accommodating non-deterministic operators, interactions with external systems, and arbitrary custom code. It is non-blocking, allowing failed operators to recover independently without interrupting other active operators, thereby leveraging data parallelization, and it facilitates dynamic scaling of operators during pipeline execution. Performance evaluations, conducted within the SAP Data Intelligence system, compare LOG.io with the Asynchronous Barrier Snapshotting (ABS) protocol, originally implemented in Flink. Our experiments show that when there are straggler operators in a data pipeline and the throughput of events is moderate (e.g., 1 event every 100 ms), LOG.io performs as well as ABS during normal processing and outperforms ABS during recovery. Otherwise, ABS performs better than LOG.io for both normal processing and recovery. However, we show that in these cases, data parallelization can largely reduce the overhead of LOG.io while ABS does not improve. Finally, we show that the overhead of data lineage capture, at the granularity of the event and between any two operators in a pipeline, is marginal, with less than 1.5% in all our experiments.

Problem

• Designs a system for rollback recovery and data lineage capture in distributed pipelines
• Addresses non-deterministic operators and external interactions in serverless architectures
• Evaluates performance against ABS protocol under varying throughput conditions

Innovation

• Log-based rollback recovery protocol for distributed pipelines
• Non-blocking independent operator recovery during pipeline execution
• Fine-grain data lineage capture with marginal overhead