ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL

2024-12-13

International Conference on Learning Representations

AI Summary

Open-source large language models (LLMs) exhibit weak generalization and severe SQL hallucination in Text-to-SQL tasks. Method: We propose a robust multi-task fine-tuning and collaborative reasoning framework. Specifically, we introduce the first multi-task supervised fine-tuning (SFT) that jointly models schema linking, noise correction, and SQL continuation. We further design Multi-Task Collaborative Prompting (MCP) to explicitly capture inter-task dependencies and suppress hallucination. Combined with synthetic data augmentation and open-LLM adaptation techniques, our approach enhances robustness against complex schemas and noisy inputs. Contribution/Results: Extensive experiments across eight mainstream open-source LLMs and five benchmarks—including Spider and BIRD—demonstrate consistent and significant improvements over prior methods, achieving state-of-the-art performance in the Text-to-SQL domain. Our framework delivers a practical, reliable solution for open-scenario Text-to-SQL applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Despite the significant advancements in Text-to-SQL (Text2SQL) facilitated by large language models (LLMs), the latest state-of-the-art techniques are still trapped in the in-context learning of closed-source LLMs (e.g., GPT-4), which limits their applicability in open scenarios. To address this challenge, we propose a novel RObust mUltitask Tuning and collaboration mEthod (ROUTE) to improve the comprehensive capabilities of open-source LLMs for Text2SQL, thereby providing a more practical solution. Our approach begins with multi-task supervised fine-tuning (SFT) using various synthetic training data related to SQL generation. Unlike existing SFT-based Text2SQL methods, we introduced several additional SFT tasks, including schema linking, noise correction, and continuation writing. Engaging in a variety of SQL generation tasks enhances the model's understanding of SQL syntax and improves its ability to generate high-quality SQL queries. Additionally, inspired by the collaborative modes of LLM agents, we introduce a Multitask Collaboration Prompting (MCP) strategy. This strategy leverages collaboration across several SQL-related tasks to reduce hallucinations during SQL generation, thereby maximizing the potential of enhancing Text2SQL performance through explicit multitask capabilities. Extensive experiments and in-depth analyses have been performed on eight open-source LLMs and five widely-used benchmarks. The results demonstrate that our proposal outperforms the latest Text2SQL methods and yields leading performance.

Problem

• Enhancing open-source LLMs for Text-to-SQL via multitask fine-tuning
• Reducing SQL hallucinations through collaborative multitask prompting
• Improving Text-to-SQL performance without relying on closed-source LLMs

Innovation

• Multi-task supervised fine-tuning with synthetic data
• Additional SFT tasks like schema linking and noise correction
• Multitask Collaboration Prompting to reduce SQL hallucinations