JudgeSQL: Reasoning over SQL Candidates with Weighted Consensus Tournament

2025-10-17

AI Summary

Candidate SQL query selection in text-to-SQL remains unreliable—existing methods (e.g., self-consistency, top-N decoding) rely on shallow signals, leading to inconsistent scoring, brittle reasoning, and poor discrimination among semantically similar queries. Method: This paper proposes a reasoning-augmented weighted consensus tournament framework. Contributions/Results: (1) An interpretable, reasoning-based SQL judge model trained via reinforcement learning with verifiable execution feedback; (2) A weighted consensus mechanism incorporating generator confidence to enhance fine-grained semantic discrimination; (3) Integration of reasoning trajectory distillation and test-time expansion for improved robustness. Evaluated on the BIRD benchmark, the framework achieves significant gains in SQL selection accuracy, demonstrates strong cross-scale generalization, and maintains robust performance across diverse SQL generators.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Text-to-SQL is a pivotal task that bridges natural language understanding and structured data access, yet it remains fundamentally challenging due to semantic ambiguity and complex compositional reasoning. While large language models (LLMs) have greatly advanced SQL generation though prompting, supervised finetuning and reinforced tuning, the shift toward test-time scaling exposes a new bottleneck: selecting the correct query from a diverse candidate pool. Existing selection approaches, such as self-consistency or best-of-$N$ decoding, provide only shallow signals, making them prone to inconsistent scoring, fragile reasoning chains, and a failure to capture fine-grained semantic distinctions between closely related SQL candidates. To this end, we introduce JudgeSQL, a principled framework that redefines SQL candidate selection through structured reasoning and weighted consensus tournament mechanism. JudgeSQL develops a reasoning-based SQL judge model that distills reasoning traces with reinforcement learning guided by verifiable rewards, enabling accurate and interpretable judgments. Building on this, a weighted consensus tournament integrates explicit reasoning preferences with implicit generator confidence, yielding selections that are both more reliable and more efficient. Extensive experiments on the BIRD benchmark demonstrate that JudgeSQL exhibits superior SQL judgment capabilities and good cross-scale generalization and robustness to generator capacity.

Problem

• Selecting correct SQL queries from diverse candidate pools
• Overcoming inconsistent scoring and fragile reasoning chains
• Addressing semantic ambiguity in text-to-SQL generation

Innovation

• Reinforcement learning distills reasoning traces
• Weighted consensus integrates reasoning preferences
• Tournament mechanism combines explicit and implicit signals