A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL

2024-11-13

AI Summary

This work addresses the low quality and poor diversity of SQL generation by large language models (LLMs) in natural language-to-SQL (NL2SQL) tasks. To this end, we propose XiYan-SQL, a multi-generator ensemble framework. Methodologically: (1) we introduce M-Schema, a structure-aware semi-structured schema representation; (2) we integrate supervised fine-tuning with named entity recognition (NER)-guided in-context learning (ICL) to produce high-quality candidate SQL queries; and (3) we design an error-aware refiner and a fine-grained multi-candidate ranking model, forming a three-stage pipeline—generation, refinement, and selection. Our key innovations include the first multi-generator coordination mechanism and an NER-driven ICL exemplar selection strategy. Extensive evaluation on four benchmarks—Bird, Spider, SQL-Eval, and NL2GQL—yields execution accuracies of 75.63%, 89.65%, 69.86%, and 41.20%, respectively, surpassing all existing methods.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

To tackle the challenges of large language model performance in natural language to SQL tasks, we introduce XiYan-SQL, an innovative framework that employs a multi-generator ensemble strategy to improve candidate generation. We introduce M-Schema, a semi-structured schema representation method designed to enhance the understanding of database structures. To enhance the quality and diversity of generated candidate SQL queries, XiYan-SQL integrates the significant potential of in-context learning (ICL) with the precise control of supervised fine-tuning. On one hand, we propose a series of training strategies to fine-tune models to generate high-quality candidates with diverse preferences. On the other hand, we implement the ICL approach with an example selection method based on named entity recognition to prevent overemphasis on entities. The refiner optimizes each candidate by correcting logical or syntactical errors. To address the challenge of identifying the best candidate, we fine-tune a selection model to distinguish nuances of candidate SQL queries. The experimental results on multiple dialect datasets demonstrate the robustness of XiYan-SQL in addressing challenges across different scenarios. Overall, our proposed XiYan-SQL achieves the state-of-the-art execution accuracy of 75.63% on Bird benchmark, 89.65% on the Spider test set, 69.86% on SQL-Eval, 41.20% on NL2GQL. The proposed framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods.

Problem

• Improve natural language to SQL conversion
• Enhance database schema understanding
• Optimize SQL query candidate selection

Innovation

• Multi-generator ensemble strategy
• Semi-structured schema representation
• In-context learning integration