Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

2024-12-21

arXiv.org

AI Summary

Existing text-to-SQL approaches struggle with ambiguous, cross-turn, and non-SQL-answering queries in multi-turn dialogues, yielding unreliable responses. This work introduces MMSQL—a novel benchmark and a multi-agent enhanced framework. First, we construct MMSQL, the first structured evaluation suite supporting multi-turn interactions and diverse question types, including non-SQL-answering ones. Second, we propose a specialized multi-agent architecture that jointly models question classification, dialogue state tracking, and SQL generation, enabling dynamic adaptation of response strategies based on predicted question type. Experiments demonstrate substantial improvements in accuracy and robustness of LLMs across ambiguity, cross-turn dependencies, and mixed-intent scenarios. The MMSQL dataset and source code are publicly released to advance research in conversational database interfaces.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Modeling other Agents

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Community question answering

Abstract

Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries. Our dataset and code are publicly available at https://mcxiaoxiao.github.io/MMSQL.

Problem

• Evaluating LLMs for multi-turn text-to-SQL with diverse question types
• Addressing unreliable responses in SQL generation for ambiguous questions
• Enhancing LLM performance via multi-agent question classification and strategies

Innovation

• MMSQL test suite evaluates LLMs for SQL generation
• Multi-agent framework identifies question types dynamically
• Handles diverse conversational queries with specialized agents