Multi-Turn Interactions for Text-to-SQL with Large Language Models

2024-08-09

Proceedings of the 34th ACM International Conference on Information and Knowledge Management

AI Summary

To address the low parsing efficiency, opaque interaction, and poor generalization of large language models (LLMs) in Text-to-SQL for wide-table scenarios, this paper proposes Interactive-T2S—a framework enabling iterative human-AI collaboration wherein the LLM directly interacts with the database to generate SQL progressively and transparently. Its core contributions are: (1) four generic, schema-agnostic database interaction tools that support cross-schema generalization; and (2) a structured, example-driven stepwise reasoning paradigm integrating dynamic context construction and chain-of-thought prompting. Evaluated on Spider and BIRD (including its variants), Interactive-T2S achieves new state-of-the-art performance on the BIRD leaderboard under the non-oracle setting, significantly improving both query accuracy on wide-table schemas and the interpretability of human-system interaction.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Intelligent Query Processing
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study explores text-to-SQL parsing by leveraging the powerful reasoning capabilities of large language models (LLMs). Despite recent advancements, existing LLM-based methods are still inefficient and struggle to handle cases with wide tables effectively. Furthermore, current interaction-based approaches either lack a step-by-step, interpretable SQL generation process or fail to provide a universally applicable interaction design. To address these challenges, we introduce Interactive-T2S, a framework that generates SQL queries through direct interactions with databases. This framework includes four general tools that facilitate proactive and efficient information retrieval by the LLM. Additionally, we have developed detailed exemplars to demonstrate the step-wise reasoning processes within our framework. Our approach achieves advanced performance on the Spider and BIRD datasets as well as their variants. Notably, we obtain state-of-the-art results on the BIRD leaderboard under the setting without oracle knowledge, demonstrating the effectiveness of our method.

Problem

• Addresses inefficient SQL generation with wide tables
• Provides step-by-step interpretable SQL query generation
• Creates universally applicable database interaction framework

Innovation

• Interactive-T2S framework for direct database interactions
• Four general tools enabling proactive information retrieval
• Step-wise reasoning exemplars for interpretable SQL generation