Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

2026-07-07

AI Summary

This work addresses the limitation of existing text-to-SQL benchmarks, which support only conventional SQL and are thus inadequate for evaluating models’ ability to generate AI-native SQL that leverages large language model capabilities. To bridge this gap, the study introduces the first benchmark specifically designed for AI-native SQL, encompassing six categories of AI functions and 125 real-world databases. The authors employ an agent-based rewriting pipeline to reformulate original tasks into queries incorporating AI functions, while simultaneously refining both natural language instructions and target queries to clarify intent. A multi-turn temporally decoupled execution protocol is introduced to ensure result stability, and systematic evaluation is conducted on the Snowflake platform with integrated AI functions. Experiments reveal that the strongest closed-source model achieves 67–70% execution accuracy, while the best open-source model reaches 58.1%; notably, simplified agent architectures outperform traditional complex frameworks in AI-native settings.

Abstract

Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries. Yet existing text-to-SQL benchmarks evaluate only conventional SQL and provide no signal on whether models can generate such AI-native SQL. We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform. Starting from an existing enterprise text-to-SQL benchmark, we construct Spider 2.0-AIFunc through an agent-based pipeline that rewrites source tasks into AI-native form, simultaneously transforming target queries and refining natural language instructions to make the intended AI-native solution explicit and reduce ambiguity. All instances pass a multi-round repeated execution protocol across temporally separated windows to confirm result stability before release. Evaluating ten state-of-the-art language models, we find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%, a gap driven primarily by errors in predicate specification, schema grounding, and AI function parameterization. Agent frameworks designed for traditional text-to-SQL challenges, such as schema retrieval and relevant table selection, do not transfer effectively to AI-native SQL: a minimal agent setup consistently matches or outperforms more elaborate alternatives, suggesting that the strategies these frameworks employ are less critical in this setting. Data are available at https://github.com/Leolty/Spider2-AIFunc .

Problem

• text-to-SQL
• AI-native SQL
• benchmark
• large language models
• cloud data platforms

Innovation

• AI-native SQL
• text-to-SQL benchmark
• LLM-as-SQL-function
• agent-based rewriting
• execution stability