In-Memory Indexing and Querying of Provenance in Data Preparation Pipelines

2025-11-05

AI Summary

To address the challenges of fine-grained provenance capture and efficient querying in data preparation workflows, this paper proposes a tensor-based in-memory indexing mechanism. The method innovatively integrates both backward- and forward-looking provenance information, employing an enhanced tensor model to explicitly encode record-level and attribute-level input–output mappings, thereby enabling expressive lineage analysis. Unlike conventional approaches, its memory-efficient index design substantially reduces storage overhead while accelerating diverse provenance queries—including origin tracing, impact analysis, and dependency exploration. Experimental evaluation on real-world and synthetic datasets demonstrates that our approach consistently outperforms state-of-the-art baselines in both query latency and memory footprint. The proposed solution thus provides scalable, high-performance provenance support for critical downstream tasks such as debugging, model interpretability, fairness auditing, and data quality diagnostics.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Machine Learning: Matrix & Tensor Methods
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

Data provenance has numerous applications in the context of data preparation pipelines. It can be used for debugging faulty pipelines, interpreting results, verifying fairness, and identifying data quality issues, which may affect the sources feeding the pipeline execution. In this paper, we present an indexing mechanism to efficiently capture and query pipeline provenance. Our solution leverages tensors to capture fine-grained provenance of data processing operations, using minimal memory. In addition to record-level lineage relationships, we provide finer granularity at the attribute level. This is achieved by augmenting tensors, which capture retrospective provenance, with prospective provenance information, drawing connections between input and output schemas of data processing operations. We demonstrate how these two types of provenance (retrospective and prospective) can be combined to answer a broad range of provenance queries efficiently, and show effectiveness through evaluation exercises using both real and synthetic data.

Problem

• Efficiently capturing and querying data pipeline provenance
• Combining retrospective and prospective provenance for queries
• Minimizing memory usage while tracking fine-grained lineage

Innovation

• In-memory indexing for efficient provenance capture
• Tensor-based fine-grained provenance with minimal memory
• Combining retrospective and prospective provenance for queries