ROSE: An Intent-Centered Evaluation Metric for NL2SQL

2026-04-14

AI Summary

Existing NL2SQL evaluation metrics, such as execution accuracy, are sensitive to syntactic variations, overlook question ambiguity, and are vulnerable to annotation errors, thereby failing to accurately capture a model’s understanding of user intent. To address this, this work proposes ROSE—a user-intent-centered evaluation framework that introduces a novel reference-free, adversarial Prover-Refuter cascade architecture: the Prover verifies whether the generated SQL semantically answers the question correctly, while the Refuter performs adversarial validation using the reference SQL as a counterexample. Leveraging this approach, we construct ROSE-VEC, an expert-aligned verification set. Experimental results demonstrate that ROSE achieves nearly a 24% improvement in agreement with human judgment (Cohen’s Kappa) over the next-best metric and yields key insights when re-evaluating 19 representative NL2SQL methods.

Technology Category

• Natural Language Processing: Question Answering
• Multiagent Systems: Adversarial Agents
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Execution Accuracy (EX), the widely used metric for evaluating the effectiveness of Natural Language to SQL (NL2SQL) solutions, is becoming increasingly unreliable. It is sensitive to syntactic variation, ignores that questions may admit multiple interpretations, and is easily misled by erroneous ground-truth SQL. To address this, we introduce ROSE, an intent-centered metric that focuses on whether the predicted SQL answers the question, rather than consistency with the ground-truth SQL under the reference-dependent paradigm. ROSE employs an adversarial Prover-Refuter cascade: SQL Prover assesses the semantic correctness of a predicted SQL against the user's intent independently, while Adversarial Refuter uses the ground-truth SQL as evidence to challenge and refine this judgment. On our expert-aligned validation set ROSE-VEC, ROSE achieves the best agreement with human experts, outperforming the next-best metric by nearly 24% in Cohen's Kappa. We also conduct a largescale re-evaluation of 19 NL2SQL methods, revealing four valuable insights. We release ROSE and ROSE-VEC to facilitate more reliable NL2SQL research.

Problem

• NL2SQL
• Evaluation Metric
• Execution Accuracy
• Semantic Interpretation
• Ground-truth SQL

Innovation

• intent-centered evaluation
• NL2SQL
• adversarial Prover-Refuter
• execution accuracy
• semantic correctness