🤖 AI Summary
This work addresses the high computational cost and insufficient engineering optimization in inclusion dependency discovery by efficiently implementing the Spider and Faida algorithms within the Desbordante system. Through implementation-level optimizations—including parallelization, SIMD vectorization, custom hash tables, and data caching—it systematically integrates algorithmic design with engineering practice for the first time. Specifically, a memory-efficient, highly parallel variant is introduced for the classic Spider algorithm, while four performance-enhancing optimizations are incorporated into the state-of-the-art approximate algorithm Faida. Compared to the Java-based Metanome framework, the optimized implementations achieve up to a 5× speedup for Spider and an 8× speedup for Faida, accompanied by substantially reduced memory consumption.
📝 Abstract
Inclusion dependency is a relation between attributes of tables that indicates possible Primary Key-Foreign Key references. Automatic discovery of inclusion dependencies is a relevant problem for both academic and industrial communities. The core concern for this problem is the efficiency of discovery process, since it is a computationally expensive task. However, existing studies only address the algorithmic side, while leaving out the implementation aspect. At the same time, engineering details are at least as important as the algorithmic ones for achieving good performance.
In this paper, we describe techniques for efficient implementation of two algorithms for discovery of inclusion dependencies - Spider and Faida. The first one is a classic algorithm whose ideas lie in the foundation of many other inclusion dependency discovery algorithms. We propose an efficient parallelization technique, which greatly speeds up the algorithm while simultaneously reducing its memory consumption. The second one is the state-of-the-art approximate algorithm, which we approach by applying four types of optimizations: data buffering, SIMD-enabled execution, careful hash-table selection and parallelization.
In order to experimentally evaluate our techniques, we have implemented these algorithms in Desbordante - an open-source science-intensive data profiler written in C++. For Spider, we have evaluated several different options, and in case of Faida we have demonstrated that all our optimization techniques yield results. We also compared our implementations with Metanome - a Java-based data profiler. Overall, we report up to 5x improvement in terms of run time reduction for Spider and up to 8x for Faida.