MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation

2024-10-16

arXiv.org

AI Summary

To address the limited accessibility, high privacy risks, and substantial inference latency of large closed-source models (e.g., GPT-4) in text-to-SQL tasks, this work proposes a lightweight, open-source alternative. Methodologically, we introduce the first multi-sample joint critique framework: leveraging small open-source LLMs (e.g., Phi-3, TinyLlama), it generates multiple candidate SQL queries, performs execution metadata–driven joint critique—using signals such as empty results and syntax errors—and applies lightweight score fusion, all without requiring additional annotations or fine-tuning. Our contributions include significantly enhanced accuracy and generalization for small LLMs, achieving state-of-the-art performance among open-source models on benchmarks including BIRD and Spider—matching GPT-4 Turbo’s effectiveness while reducing inference cost by over 90% and delivering end-to-end latency under 800 ms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Text-to-SQL generation enables non-experts to interact with databases via natural language. Recent advances rely on large closed-source models like GPT-4 that present challenges in accessibility, privacy, and latency. To address these issues, we focus on developing small, efficient, and open-source text-to-SQL models. We demonstrate the benefits of sampling multiple candidate SQL generations and propose our method, MSc-SQL, to critique them using associated metadata. Our sample critiquing model evaluates multiple outputs simultaneously, achieving state-of-the-art performance compared to other open-source models while remaining competitive with larger models at a much lower cost. Full code can be found at https://github.com/layer6ai-labs/msc-sql.

Problem

• Develop efficient open-source text-to-SQL models
• Critique multiple SQL generations using metadata
• Achieve state-of-the-art performance at low cost

Innovation

• Small open-source text-to-SQL models
• Multiple candidate SQL generations sampling
• Critiquing using associated metadata