LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models

2025-04-03

AI Summary

Open-source large language models (LLMs) underperform on complex NL2SQL tasks due to implicit user intent and semantic gaps between natural language queries and database schemas. Method: This paper proposes an Abstract Syntax Tree (AST)-guided task decomposition framework comprising: (1) a novel three-stage AST-based decomposition mechanism that explicitly models SQL syntactic structure and semantic composition; (2) margin-aware Direct Preference Optimization (DPO), leveraging AST-derived margin signals to refine SQL generation; and (3) an adaptive in-context example retrieval strategy to dynamically strengthen decomposition reasoning. Results: Evaluated on Spider and BIRD benchmarks, the framework achieves GPT-4-level accuracy using only a 7B open-source LLM, with significantly improved inference speed. It substantially reduces reliance on proprietary models and large-scale human-annotated datasets, advancing efficient, scalable, and interpretable NL2SQL.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Natural Language to SQL (NL2SQL) has emerged as a critical task for enabling seamless interaction with databases. Recent advancements in Large Language Models (LLMs) have demonstrated remarkable performance in this domain. However, existing NL2SQL methods predominantly rely on closed-source LLMs leveraging prompt engineering, while open-source models typically require fine-tuning to acquire domain-specific knowledge. Despite these efforts, open-source LLMs struggle with complex NL2SQL tasks due to the indirect expression of user query objectives and the semantic gap between user queries and database schemas. Inspired by the application of reinforcement learning in mathematical problem-solving to encourage step-by-step reasoning in LLMs, we propose LearNAT (Learning NL2SQL with AST-guided Task Decomposition), a novel framework that improves the performance of open-source LLMs on complex NL2SQL tasks through task decomposition and reinforcement learning. LearNAT introduces three key components: (1) a Decomposition Synthesis Procedure that leverages Abstract Syntax Trees (ASTs) to guide efficient search and pruning strategies for task decomposition, (2) Margin-aware Reinforcement Learning, which employs fine-grained step-level optimization via DPO with AST margins, and (3) Adaptive Demonstration Reasoning, a mechanism for dynamically selecting relevant examples to enhance decomposition capabilities. Extensive experiments on two benchmark datasets, Spider and BIRD, demonstrate that LearNAT enables a 7B-parameter open-source LLM to achieve performance comparable to GPT-4, while offering improved efficiency and accessibility.

Problem

• Enhancing open-source LLMs for complex NL2SQL tasks
• Bridging semantic gap between queries and databases
• Improving efficiency via AST-guided task decomposition

Innovation

• AST-guided task decomposition for NL2SQL
• Margin-aware reinforcement learning optimization
• Adaptive demonstration reasoning for decomposition