Auto-Prep: Holistic Prediction of Data Preparation Steps for Self-Service Business Intelligence

2025-04-15

AI Summary

This paper addresses the inefficiency, tight coupling, and heavy manual programming burden in table joining and transformation during the data preparation phase of self-service business intelligence (BI). Drawing insights from an analysis of 2,000 real-world BI projects, we propose— for the first time—the joint modeling of these two interdependent tasks. Our method introduces a novel graph-based model inspired by Steiner trees, which unifies join and transformation prediction within a single framework. It integrates graph neural networks, Steiner tree approximation algorithms, BI workflow pattern mining, and empirically grounded rule-augmented reinforcement learning. The approach provides theoretical guarantees on solution quality and overcomes the limitations of conventional isolated modeling. Evaluated on real BI datasets, our method achieves over 70% accuracy—significantly outperforming state-of-the-art domain-specific algorithms and large language models including GPT-4.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Intelligent Query Processing
• Reasoning under Uncertainty: Graphical Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Business Intelligence (BI) plays a critical role in empowering modern enterprises to make informed data-driven decisions, and has grown into a billion-dollar business. Self-service BI tools like Power BI and Tableau have democratized the ``dashboarding'' phase of BI, by offering user-friendly, drag-and-drop interfaces that are tailored to non-technical enterprise users. However, despite these advances, we observe that the ``data preparation'' phase of BI continues to be a key pain point for BI users today. In this work, we systematically study around 2K real BI projects harvested from public sources, focusing on the data-preparation phase of the BI workflows. We observe that users often have to program both (1) data transformation steps and (2) table joins steps, before their raw data can be ready for dashboarding and analysis. A careful study of the BI workflows reveals that transformation and join steps are often intertwined in the same BI project, such that considering both holistically is crucial to accurately predict these steps. Leveraging this observation, we develop an Auto-Prep system to holistically predict transformations and joins, using a principled graph-based algorithm inspired by Steiner-tree, with provable quality guarantees. Extensive evaluations using real BI projects suggest that Auto-Prep can correctly predict over 70% transformation and join steps, significantly more accurate than existing algorithms as well as language-models such as GPT-4.

Problem

• Predicts data transformation steps in BI workflows
• Identifies table joins needed for BI data preparation
• Holistically combines transformations and joins for accuracy

Innovation

• Holistic prediction of BI data preparation steps
• Graph-based algorithm inspired by Steiner-tree
• Automated transformation and join step prediction