EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing

2025-03-28

AI Summary

To address the high computational cost and poor scalability of Text-to-SQL systems stemming from reliance on large language models (LLMs), this paper proposes the first complexity-aware dynamic routing framework. Our method introduces Token Elasticity of Performance (TEP)—a novel metric quantifying the trade-off between accuracy and token cost—and integrates multi-router coordination (including Qwen2.5-0.5B-DPO), query complexity estimation, and adaptive scheduling between lightweight and heavyweight SQL generation models to route each query to its optimal execution path. Evaluated on the BIRD development set, our framework reduces total token consumption by over 40% compared to uniformly invoking the strongest model, doubles TEP, and maintains zero degradation in SQL execution accuracy. These results demonstrate significant improvements in cost efficiency and resource sustainability without compromising performance.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Text-to-SQL automatically translates natural language queries to SQL, allowing non-technical users to retrieve data from databases without specialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQL approaches on leaderboards, their unsustainable computational costs--often overlooked--stand as the"elephant in the room"in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we exploratively propose EllieSQL, a complexity-aware routing framework that assigns queries to suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity of Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in our study, EllieSQL with the Qwen2.5-0.5B-DPO router reduces token use by over 40% without compromising performance on Bird development set, achieving more than a 2x boost in TEP over non-routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL.

Problem

• Reducing computational costs in Text-to-SQL systems
• Optimizing query routing based on complexity estimation
• Improving cost-efficiency without sacrificing performance

Innovation

• Complexity-aware routing for Text-to-SQL
• Token Elasticity of Performance metric
• Efficient query assignment to SQL pipelines