Exploring the Use of LLMs for SQL Equivalence Checking

2024-12-07

arXiv.org

AI Summary

Automated equivalence checking for complex SQL queries—critical for database education and query optimizer debugging—lacks efficient, reliable solutions. Method: This work pioneers the systematic evaluation of large language models (LLMs) for high-accuracy SQL equivalence judgment without formal proofs. We propose a prompt engineering framework grounded in unoptimized logical query plans, integrating SQL parsing, multi-strategy prompting, synthetic data augmentation, and task-specific supervised fine-tuning to bridge the performance gap between smaller LLMs and GPT-class models. Contribution/Results: Our approach achieves ≈100% accuracy on equivalent SQL pairs and 70% on non-equivalent pairs, while generating human-readable step-by-step reasoning and concrete counterexamples. It extends support beyond restricted SQL subsets—unlike conventional methods—and establishes a practical, interpretable, LLM-driven paradigm for database education and system debugging.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Equivalence checking of two SQL queries is an intractable problem encountered in diverse contexts ranging from grading student submissions in a DBMS course to debugging query rewriting rules in an optimizer, and many more. While a lot of progress has been made in recent years in developing practical solutions for this problem, the existing methods can handle only a small subset of SQL, even for bounded equivalence checking. They cannot support sophisticated SQL expressions one encounters in practice. At the same time, large language models (LLMs) -- such as GPT-4 -- have emerged as power generators of SQL from natural language specifications. This paper explores whether LLMs can also demonstrate the ability to reason with SQL queries and help advance SQL equivalence checking. Towards this, we conducted a detailed evaluation of several LLMs over collections with SQL pairs of varying levels of complexity. We explored the efficacy of different prompting techniques, the utility of synthetic examples&explanations, as well as logical plans generated by query parsers. Our main finding is that with well-designed prompting using an unoptimized SQL Logical Plan, LLMs can perform equivalence checking beyond the capabilities of current techniques, achieving nearly 100% accuracy for equivalent pairs and up to 70% for non-equivalent pairs of SQL queries. While LLMs lack the ability to generate formal proofs, their synthetic examples and human-readable explanations offer valuable insights to students (&instructors) in a classroom setting and to database administrators (DBAs) managing large database installations. Additionally, we also show that with careful fine-tuning, we can close the performance gap between smaller (and efficient) models and larger models such as GPT, thus paving the way for potential LLM-integration in standalone data processing systems.

Problem

• LLMs address SQL query equivalence checking limitations
• Benchmark SQLEquiQuest tests real-world SQL complexity
• LLMs show bias in non-equivalent SQL pair predictions

Innovation

• LLMs for SQL equivalence checking
• Novel benchmark SQLEquiQuest introduced
• Prompting strategies with logical plans