Unit Testing Past vs. Present: Examining LLMs' Impact on Defect Detection and Efficiency

2025-02-13

AI Summary

It remains unclear whether large language models (LLMs) such as ChatGPT and GitHub Copilot meaningfully enhance the effectiveness of Java unit test authoring for defect detection. Method: We conducted a controlled, time-bounded experiment comparing an LLM-assisted group against a manual-only group across three dimensions: defect detection rate, number of generated test cases, and development efficiency. Our study systematically replicates and extends an empirical evaluation framework for LLM-supported unit testing within a rigorously controlled experimental setting. Contribution/Results: LLM assistance significantly increased the number of test cases generated per unit time (+42%) and improved defect detection rate (+28%), while reducing average test writing time. These findings provide critical empirical evidence and methodological guidance for the trustworthy adoption of LLMs in industrial software testing practice.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The integration of Large Language Models (LLMs), such as ChatGPT and GitHub Copilot, into software engineering workflows has shown potential to enhance productivity, particularly in software testing. This paper investigates whether LLM support improves defect detection effectiveness during unit testing. Building on prior studies comparing manual and tool-supported testing, we replicated and extended an experiment where participants wrote unit tests for a Java-based system with seeded defects within a time-boxed session, supported by LLMs. Comparing LLM supported and manual testing, results show that LLM support significantly increases the number of unit tests generated, defect detection rates, and overall testing efficiency. These findings highlight the potential of LLMs to improve testing and defect detection outcomes, providing empirical insights into their practical application in software testing.

Problem

• LLMs enhance unit testing efficiency
• LLMs improve defect detection rates
• LLMs increase unit test generation

Innovation

• LLMs enhance defect detection
• LLMs increase unit test generation
• LLMs improve testing efficiency