NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

2026-04-13

AI Summary

Existing NL2SQL systems lack a modular, fine-grained evaluation framework, making it difficult to comprehensively assess their performance and limitations. This work proposes the first modular benchmarking framework that decomposes the NL2SQL pipeline into three core components: Schema Selection, Candidate Generation, and Query Revision, each accompanied by tailored fine-grained metrics. Leveraging a multi-agent architecture, the framework enables flexible and systematic evaluation. Comprehensive experiments on BIRD and ScienceBenchmark across ten open-source methods reveal pervasive issues of insufficient accuracy and excessive computational overhead. Furthermore, the study uncovers erroneous gold-standard SQL annotations and flaws in existing evaluation protocols. By establishing a reproducible evaluation paradigm and identifying clear directions for improvement, this work lays a foundation for more rigorous and transparent progress in NL2SQL research.

Technology Category

• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large language models (LLMs) have greatly improved NL2SQL algorithms, their rapid development outpaces systematic evaluation, leaving a critical gap in understanding their effectiveness, efficiency, and limitations. To this end, we present NL2SQLBench, the first modular evaluation and benchmarking framework for LLM-enabled NL2SQL approaches. Specifically, we dissect NL2SQL systems into three core modules: Schema Selection, Candidate Generation, and Query Revision. For each module, we comprehensively review existing strategies and propose novel fine-grained metrics that systematically quantify module-level effectiveness and efficiency. We further implement these metrics in a flexible multi-agent framework, allowing configurable benchmarking across diverse NL2SQL approaches. Leveraging NL2SQLBench, we rigorously evaluate ten representative open-source methods on two datasets, the BIRD development set and the ScienceBenchmark development set, using two LLMs, DeepSeek-V3 and GPT-4o mini. We systematically assess each approach across the three core modules and evaluate multiple critical performance dimensions. Our evaluation reveals significant gaps in existing NL2SQL methods, highlighting not only substantial room for accuracy improvements but also the significant computational inefficiency, which severely hampers real-world adoption. Furthermore, our analysis identifies critical shortcomings in current benchmark datasets and evaluation rules, emphasizing issues such as inaccurate gold SQL annotations and limitations in existing evaluation rules. By synthesizing these insights into a unified benchmarking, our study establishes a clear reference point for fair comparison and serves as essential guidance for future targeted innovation in NL2SQL technology.

Problem

• NL2SQL
• large language models
• benchmarking
• evaluation framework
• systematic assessment

Innovation

• modular benchmarking
• NL2SQL
• large language models
• fine-grained evaluation
• multi-agent framework