On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems

2026-01-13

AI Summary

This study addresses the reliability challenges posed by flakiness in test cases generated by large language models (LLMs), which can compromise defect detection. For the first time, it systematically quantifies the prevalence and root causes of flakiness in tests produced by GPT-4o and Mistral-Large-Instruct-2407 across four database systems: SAP HANA, DuckDB, MySQL, and SQLite. Through manual analysis and cross-system comparative experiments, the authors find that LLM-generated tests exhibit slightly higher flakiness than existing human-written tests. Notably, 63% of flaky cases stem from implicit assumptions about unordered collections—termed “non-guaranteed order dependencies.” Furthermore, flakiness can propagate from existing tests to newly generated ones via contextual prompts, a phenomenon particularly pronounced in closed-source systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Flaky tests are a common problem in software testing. They produce inconsistent results when executed multiple times on the same code, invalidating the assumption that a test failure indicates a software defect. Recent work on LLM-based test generation has identified flakiness as a potential problem with generated tests. However, its prevalence and underlying causes are unclear. We examined the flakiness of LLM-generated tests in the context of four relational database management systems: SAP HANA, DuckDB, MySQL, and SQLite. We amplified test suites with two LLMs, GPT-4o and Mistral-Large-Instruct-2407, to assess the flakiness of the generated test cases. Our results suggest that generated tests have a slightly higher proportion of flaky tests compared to existing tests. Based on a manual inspection, we found that the most common root cause of flakiness was the reliance of a test on a certain order that is not guaranteed ("unordered collection"), which was present in 72 of 115 flaky tests (63%). Furthermore, both LLMs transferred the flakiness from the existing tests to the newly generated tests via the provided prompt context. Our experiments suggest that flakiness transfer is more prevalent in closed-source systems such as SAP HANA than in open-source systems. Our study informs developers on what types of flakiness to expect from LLM-generated tests. It also highlights the importance of providing LLMs with tailored context when employing LLMs for test generation.

Problem

• flaky tests
• LLM-generated tests
• test flakiness
• database management systems
• software testing

Innovation

• flaky tests
• LLM-generated tests
• test flakiness transfer
• unordered collection
• database testing