ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Format Restriction, and Column Exploration

2025-02-02

AI Summary

Existing Text-to-SQL methods struggle with enterprise-scale databases—characterized by >3,000 columns, heterogeneous SQL dialects, and analytical queries—due to limitations in instruction following, long-context modeling, self-correction, and dialect adaptation. This paper proposes the first unified framework addressing these challenges: (1) table compression to mitigate context bloat; (2) format-constrained decoding to enforce SQL syntactic and semantic validity; (3) iterative column exploration to deepen schema understanding; and (4) a two-stage self-correction mechanism combining parallel voting and CTE-guided refinement. Leveraging dialect-aware fine-tuning and parallelized inference, our model achieves new state-of-the-art execution accuracy of 26.69% on Spider 2.0-Snow and 24.50% on Spider 2.0-Lite—surpassing prior best results by over 6 percentage points—and significantly improves robustness and generalization for complex analytical query generation.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text-to-SQL systems have unlocked easier access to critical data insights by enabling natural language queries over structured databases. However, deploying such systems in enterprise environments remains challenging due to factors such as large, complex schemas (>3000 columns), diverse SQL dialects (e.g., BigQuery, Snowflake) and sophisticated query requirements (e.g., transformation, analytics). Current state-of-the-art performance on the Spider 2.0 dataset -- a benchmark built to mimic such complex environments -- remains limited at 20%. Key limitations include inadequate instruction-following, poor long-context comprehension, weak self-refinement, and insufficient dialect-specific knowledge. To address these gaps, we propose ReFoRCE (Self-Refinement Agent with Format Restriction and Column Exploration) which introduces (1) table compression to mitigate long-context limitations (2) format restriction to ensure accurate answer format, and (3) iterative column exploration for enhanced schema understanding. Additionally, it employs self-refinement pipeline consisting of (1) parallelized workflows with voting mechanisms and (2) a Common Table Expression (CTE) based refinement approach to handle unresolved cases. ReFoRCE achieves state-of-the-art results scoring 26.69 on the Spider 2.0-Snow and scoring 24.50 on the Spider 2.0-Lite tasks.

Problem

• Large-scale Database Search
• Natural Language Processing in SQL
• Adaptive Query Understanding

Innovation

• Data Table Compression
• Format Constraints
• Column Exploration