EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries

2026-06-25

AI Summary

Traditional ETL approaches rely on manually predefined schemas, making them inflexible for dynamically analyzing multi-source, heterogeneous tables in data lakes in response to natural language queries. This work proposes the first table integration framework tailored for natural language querying, which combines a lightweight graph model with Steiner tree search. Building upon a two-stage schema linking process, the framework leverages large language models to automatically generate structured views that support corresponding SQL queries. The proposed method reduces large model invocation costs to one-fifth of the original while achieving over a 30% improvement in query accuracy across four real-world datasets, significantly outperforming existing techniques.

Abstract

The diverse formats of CSV and Parquet files in data lakes pose a significant challenge to traditional ETL, which relies on data engineers to pre-define a target database schema and build a complex pipeline for data integration. Moreover, with this approach, the integrated data often cannot support various analytical needs, as the predefined schema does not necessarily satisfy the table format or join relationships required to answer unforeseen queries. To address this, we propose EcoTable, the first natural language-based data integration framework. Given a set of user-specified natural language queries, EcoTable automatically integrates the tables into a form that adequately supports the corresponding SQL queries. EcoTable achieves this by leveraging the semantic understanding and complex reasoning capabilities of LLMs. Moreover, EcoTable addresses the scalability and cost issues introduced by expensive LLM inferences with a set of novel ideas. First, EcoTable introduces a graph to represent the overall search space, where nodes represent tables and edges carry weights indicating join likelihood produced by a lightweight deep learning model. On top of this graph data structure, EcoTable designs three components to achieve our goal: (1) the table identification layer aims to identify relevant tables via a two-stage schema linking based on user queries; (2) the graph-based validation layer aims to discover significant join paths, including necessary data transformations and bridging tables, by modeling the problem as Steiner tree searches; and (3) the table transformation layer generates transformation code to implement the joins using LLMs. We construct 4 real-world benchmark datasets with more than 200 queries. Extensive experiments demonstrate that EcoTable outperforms the state-of-the-art baselines, increasing accuracy by more than 30% and cutting LLM invocation costs by 5 times.

Problem

• data lakes
• table integration
• natural language queries
• schema flexibility
• ETL

Innovation

• natural language data integration
• large language models
• data lakes
• graph-based validation
• cost-effective LLM inference