SQuaD-SQL: Efficient Text-to-SQL with Small Language Models via LLM-Guided Knowledge Distillation

2026-07-09

AI Summary

This work addresses the challenge of deploying large language models (LLMs) for Text-to-SQL tasks in resource-constrained environments due to their high computational overhead. To overcome this limitation, the authors propose a three-stage efficient training framework: first, leveraging an LLM to synthesize high-quality training data; second, applying parameter-efficient fine-tuning that is feasible on a single GPU; and third, incorporating domain adaptation strategies to enhance the performance of smaller models. Evaluated on the WikiSQL benchmark, the resulting model achieves an execution accuracy of 86.9%, substantially narrowing the performance gap with much larger models while significantly reducing inference latency and memory consumption. This approach offers a practical and lightweight solution for real-world Text-to-SQL applications under limited computational resources.

Abstract

Text-to-SQL is a fundamental task in natural language processing that enables users to interact with structured databases using natural language. While large language models (LLMs) have demonstrated remarkable performance on this task, their substantial computational requirements hinder deployment in resource-constrained settings. In this paper, we introduce SQuaD-SQL (Small-Qualified and Distilled for SQL), a novel approach that empowers small language models (SLMs) to approach the performance of LLMs on the Text-to-SQL task while significantly improving efficiency through knowledge distillation and synthetic data generation. Our method comprises three key components: (1) LLM-based synthetic data generation, where structured knowledge is extracted from LLMs via carefully designed prompting strategies; (2) parameter-efficient fine-tuning, enabling full model training on a single consumer-grade GPU; and (3) domain-adaptive fine-tuning, where domain-specific synthetic data further enhances performance in targeted domains. Experiments on the WikiSQL dataset demonstrate that SQuaD-SQL achieves an execution accuracy of 86.9% on the test set, approaching the performance of LLMs while offering faster inference and lower memory usage. These results suggest that, with proper training strategies, SLMs can serve as practical and efficient alternatives for Text-to-SQL applications in resource-limited environments.

Problem

• Text-to-SQL
• small language models
• large language models
• resource-constrained environments
• efficiency

Innovation

• knowledge distillation
• small language models
• synthetic data generation
• parameter-efficient fine-tuning
• domain-adaptive fine-tuning