Diverse Unionable Tuple Search: Novelty-Driven Discovery in Data Lakes [Technical Report]

2025-08-31

AI Summary

Existing data lake federated search methods rely on similarity matching, often retrieving tables highly redundant with the query table—thus yielding limited novelty and diversity in expanded information. To address this, we propose DUST, the first algorithm to integrate a clustering-based diversity mechanism into the joinable tuple discovery task. DUST employs a novel semantic embedding model that jointly encodes both column-level and table-level semantics, coupled with an efficient tuple encoding strategy, enabling precise identification of semantically joinable yet content-complementary tuples. Evaluated on real-world data lake benchmarks, DUST achieves over 6× speedup versus the best baseline while improving diversity metrics by at least 15%. These gains significantly enhance the information gain and practical utility of data expansion.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Intelligent Query Processing
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Semantics and Knowledge: Applications of semantic technologies for improving search, browsing, recommendation, personalization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Unionable table search techniques input a query table from a user and search for data lake tables that can contribute additional rows to the query table. The definition of unionability is generally based on similarity measures which may include similarity between columns (e.g., value overlap or semantic similarity of the values in the columns) or tables (e.g., similarity of table embeddings). Due to this and the large redundancy in many data lakes (which can contain many copies and versions of the same table), the most unionable tables may be identical or nearly identical to the query table and may contain little new information. Hence, we introduce the problem of identifying unionable tuples from a data lake that are diverse with respect to the tuples already present in a query table. We perform an extensive experimental analysis of well-known diversity algorithms applied to this novel problem and identify a gap that we address with a novel, clustering-based tuple diversity algorithm called DUST. DUST uses a novel embedding model to represent unionable tuples that outperforms other tuple representation models by at least 15 % when representing unionable tuples. Using real data lake benchmarks, we show that our diversification algorithm is more than six times faster than the most efficient diversification baseline. We also show that it is more effective in diversifying unionable tuples than existing diversification algorithms.

Problem

• Finding diverse unionable tables to avoid redundant information
• Improving novelty in data discovery from large data lakes
• Addressing redundancy issues in unionable table search techniques

Innovation

• Clustering-based algorithm for tuple diversity
• Novel embedding model for unionable tuples
• Faster and more effective diversification method