TCSR-SQL: Towards Table Content-aware Text-to-SQL with Self-retrieval

2024-07-01

CAAI Transactions on Intelligence Technology

AI Summary

In real-world Text-to-SQL tasks, natural language questions frequently contain ambiguous data keywords or reference non-existent column names in the database, causing significant performance degradation in existing large language model (LLM)-based approaches. To address this, we propose the first table-content-aware self-retrieval Text-to-SQL framework: it dynamically extracts semantic keywords via LLM in-context learning, integrates fuzzy database retrieval with knowledge-augmented schema inference to reconstruct plausible database schemas, and employs iterative “generate–execute–revise” cycles to refine SQL queries. Evaluated on our novel, manually curated benchmark of 2,115 table-content-sensitive instances, our method achieves an execution accuracy gain of over 27.8% relative to state-of-the-art methods. This work is the first to systematically resolve two core challenges in content-driven SQL generation: keyword ambiguity and schema mismatch.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Large language model‐based (LLM‐based) text‐to‐SQL methods have achieved important progress in generating SQL queries for real‐world applications. When confronted with table content‐aware questions in real‐world scenarios, ambiguous data content keywords and nonexistent database schema column names within the question lead to the poor performance of existing methods. To solve this problem, we propose a novel approach towards table content‐aware text‐to‐SQL with self‐retrieval (TCSR‐SQL). It leverages LLM's in‐context learning capability to extract data content keywords within the question and infer possible related database schema, which is used to generate Seed SQL to fuzz search databases. The search results are further used to confirm the encoding knowledge with the designed encoding knowledge table, including column names and exact stored content values used in the SQL. The encoding knowledge is sent to obtain the final Precise SQL following multi‐rounds of generation‐execution‐revision process. To validate our approach, we introduce a table‐content‐aware, question‐related benchmark dataset, containing 2115 question‐SQL pairs. Comprehensive experiments conducted on this benchmark demonstrate the remarkable performance of TCSR‐SQL, achieving an improvement of at least 27.8% in execution accuracy compared to other state‐of‐the‐art methods.

Problem

• Resolving ambiguous data content keywords in Text-to-SQL
• Handling nonexistent database schema column names in questions
• Improving SQL generation accuracy for table content-aware scenarios

Innovation

• Uses self-retrieval to extract data content keywords
• Generates seed SQL for fuzzy database searching
• Employs multi-round generation-execution-revision for precise SQL