SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications

2025-06-23

AI Summary

While large language models (LLMs) perform well on text-to-SQL translation, systematic evaluation and effective training for SQL debugging—particularly error localization and repair—in realistic database environments remain underexplored. To address this gap, we (1) introduce BIRD-CRITIC, the first SQL debugging benchmark tailored for open-source LLMs, alongside Six-Gym, a multi-dialect training environment supporting PostgreSQL, MySQL, SQLite, Oracle, SQL Server, and DuckDB; (2) propose f-Plan Boosting, a method that distills high-level debugging plans from SQL execution traces to improve training efficiency; and (3) design SQL-Rewind, a strategy that逆向 generates executable question-repair pairs, enabling supervised fine-tuning of Bird-Fixer—an intelligent agent built upon Qwen-2.5-Coder-14B. On BIRD-CRITIC-PG, Bird-Fixer achieves a 38.11% success rate, surpassing Claude-3.7-Sonnet and GPT-4.1. This work establishes the first standardized evaluation framework and strong open-source baseline for SQL debugging.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging SQL issues. To address this gap, we introduce BIRD-CRITIC, a new SQL issue debugging benchmark comprising 530 PostgreSQL tasks (BIRD-CRITIC-PG) and 570 multi-dialect tasks (BIRD-CRITIC-Multi), distilled from authentic user issues and replayed within new environments to facilitate rigorous evaluation. Baseline evaluations underscore the task's complexity, with the leading reasoning model O3-Mini achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi. Meanwhile, advancing open-source models for database tasks is crucial for empowering local development while safeguarding data privacy. Therefore, we present Six-Gym (Sql-fIX-Gym), a training environment for elevating open-source model capabilities for SQL issue debugging. This environment leverages SQL-Rewind strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose f-Plan Boosting, which extracts high-level debugging plans from SQL solutions, enabling teacher LLMs to produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, Bird-Fixer. Based on Qwen-2.5-Coder-14B, Bird-Fixer achieves 38.11% success rate on BIRD-CRITIC-PG and 29.65% on BIRD-CRITIC-Multi, surpassing leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities. The leaderboard and source code are available: https://bird-critic.github.io/

Problem

• Debugging complex SQL issues in real-world applications
• Evaluating LLMs on SQL issue debugging tasks
• Enhancing open-source models for SQL debugging capabilities

Innovation

• BIRD-CRITIC benchmark for SQL debugging evaluation
• Six-Gym training with SQL-Rewind dataset generation
• f-Plan Boosting enhances debugging trajectory efficiency