🤖 AI Summary
This study addresses the limitations of traditional functional dependencies in real-world data, where noise undermines their validity, and highlights the inadequacy of existing approximate functional dependencies (AFDs) in capturing probabilistic associations. The paper presents the first systematic investigation of probabilistic functional dependencies (pFDs), introducing an efficient pFD discovery algorithm implemented in the open-source data profiling tool Desbordante. Through rigorous theoretical analysis and empirical evaluation, the work elucidates the fundamental distinctions between pFDs and AFDs and identifies scenarios where pFDs offer clear advantages. By filling a critical gap in the empirical study of pFDs, this research demonstrates that pFDs and AFDs are not interchangeable and establishes a novel paradigm for dependency discovery in dirty data environments.
📝 Abstract
Data profiling aims to extract complex patterns from data for further analysis and use that data in domains such as data cleaning, data deduplication, anomaly detection, and many more.
Functional dependencies (FDs) are one of the most well-known patterns. However, they are poorly suited for these tasks, as real data is usually dirty, and the rigid definition of FDs does not allow algorithms to locate them. For this reason, there are several formulations aimed at relaxing FDs to support dirty data, with approximate functional dependency (AFD) being the most popular one. Another formulation is the Probabilistic Functional Dependency (pFD), which we aim to support inside Desbordante - a science-intensive, high-performance and open-source data profiling tool implemented in C++. However, pFDs are relatively poorly studied, compared to AFDs.
In this paper we study pFDs, both analytically and empirically. We start by assessing how different pFDs and AFDs are by studying cases in which pFDs have an edge over AFDs. Then, we implement the algorithm for pFD discovery, as well as study its run time and memory consumption. We also compare it with an AFD discovery algorithm. Lastly, we study the output of both algorithms to learn whether or not it is possible to use AFD discovery algorithm to get pFDs and vice versa.