Scaling Automated Database System Testing

2025-03-27

AI Summary

Existing automated database testing approaches suffer from poor cross-DBMS scalability, as they require manually tailored SQL generators for each SQL dialect, resulting in low adoption. This paper introduces SQLancer++, the first unified logical-bug detection framework supporting arbitrary DBMSs compliant with a standard SQL subset. Its core innovation is an adaptive SQL generator that dynamically infers the target DBMS’s syntactic boundaries via interactive grammar inference—eliminating manual adaptation. Integrated with a multi-DBMS-compatible testing infrastructure and lightweight, DBMS-specific oracles, SQLancer++ enables zero-configuration deployment. We validate it across 17 heterogeneous DBMSs, uncovering 195 previously unknown logical bugs; 180 have been officially confirmed and patched. SQLancer++ significantly improves the efficiency, coverage, and scalability of database reliability verification.

Technology Category

• Machine Learning: Statistical Relational/Logic Learning
• Knowledge Representation and Reasoning: Logic Programming
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recently, various automated testing approaches have been proposed that use specialized test oracles to find hundreds of logic bugs in mature, widely-used Database Management Systems (DBMSs). These test oracles require database and query generators, which must account for the often significant differences between the SQL dialects of these systems. Since it can take weeks to implement such generators, many DBMS developers are unlikely to invest the time to adopt such automated testing approaches. In short, existing approaches fail to scale to the plethora of DBMSs. In this work, we present both a vision and a platform, SQLancer++, to apply test oracles to any SQL-based DBMS that supports a subset of common SQL features. Our technical core contribution is a novel architecture for an adaptive SQL statement generator. This adaptive SQL generator generates SQL statements with various features, some of which might not be supported by the given DBMS, and then learns through interaction with the DBMS, which of these are understood by the DBMS. Thus, over time, the generator will generate mostly valid SQL statements. We evaluated SQLancer++ across 17 DBMSs and discovered a total of 195 unique, previously unknown bugs, of which 180 were fixed after we reported them. While SQLancer++ is the first major step towards scaling automated DBMS testing, various follow-up challenges remain.

Problem

• Scaling automated testing for diverse SQL DBMSs
• Reducing implementation time for SQL query generators
• Adaptive SQL generation for unsupported DBMS features

Innovation

• Adaptive SQL generator learns DBMS compatibility
• Supports any SQL-based DBMS with common features
• Automatically discovers and reports database bugs