DB-Explore: Automated Database Exploration and Instruction Synthesis for Text-to-SQL

2025-03-06

AI Summary

Existing text-to-SQL approaches exhibit limited performance on complex databases, primarily due to insufficient deep understanding of database structure and semantics. This paper proposes a database-aware lightweight alignment framework: first, constructing a graph-based database schema representation; second, leveraging GPT-4 to jointly mine structural and semantic patterns for automated schema comprehension and diverse instruction distillation; third, fine-tuning Qwen2.5-coder-7B to drastically reduce reliance on computationally expensive, closed-source large language models. Evaluated on BIRD and Spider benchmarks, our method achieves 52.1% and 84.0% execution accuracy, respectively—surpassing multiple GPT-4–driven baselines at minimal computational cost while approaching state-of-the-art performance. The core contribution lies in the first integration of database graph modeling with LLM-driven semantic mining, enabling systematic, end-to-end alignment between the model and the underlying database.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Natural Language Processing: Question Answering
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent text-to-SQL systems powered by large language models (LLMs) have demonstrated remarkable performance in translating natural language queries into SQL. However, these systems often struggle with complex database structures and domain-specific queries, as they primarily focus on enhancing logical reasoning and SQL syntax while overlooking the critical need for comprehensive database understanding. To address this limitation, we propose DB-Explore, a novel framework that systematically aligns LLMs with database knowledge through automated exploration and instruction synthesis. DB-Explore constructs database graphs to capture complex relational schemas, leverages GPT-4 to systematically mine structural patterns and semantic knowledge, and synthesizes instructions to distill this knowledge for efficient fine-tuning of LLMs. Our framework enables comprehensive database understanding through diverse sampling strategies and automated instruction generation, bridging the gap between database structures and language models. Experiments conducted on the SPIDER and BIRD benchmarks validate the effectiveness of DB-Explore, achieving an execution accuracy of 52.1% on BIRD and 84.0% on SPIDER. Notably, our open-source implementation, based on the Qwen2.5-coder-7B model, outperforms multiple GPT-4-driven text-to-SQL systems in comparative evaluations, and achieves near state-of-the-art performance with minimal computational cost.

Problem

• Enhances text-to-SQL systems for complex database structures.
• Automates database exploration and instruction synthesis for LLMs.
• Improves SQL translation accuracy on domain-specific queries.

Innovation

• Automated database graph construction for relational schemas
• GPT-4 leveraged for structural and semantic pattern mining
• Instruction synthesis for efficient LLM fine-tuning