How effective are traditional test criteria at detecting bugs in large language models generated code?

2026-09-08

AI Summary

研究探讨了传统测试标准在检测大型语言模型生成代码中的错误的有效性，使用了5个语言模型和4个基准进行实验，评估了语句覆盖、分支覆盖和变异测试三种方法。

Abstract

Test adequacy criteria are widely used to evaluate and guide software testing. Although prior research has extensively examined these criteria using human-written programs, faults, and tests, the increasing adoption of Large Language Models (LLMs) for code generation raises important questions about their effectiveness in detecting LLM-induced faults. To investigate this, we conduct an empirical study involving 5 LLMs and 4 benchmarks, simulating end-to-end workflows in which both code and tests are automatically generated. We collect 6,000+ faulty program instances and evaluate the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing. Our findings reveal several key insights. First, most faults introduced by LLMs are relatively trivial to catch. Second, the challenging faults are difficult to trigger using either traditional coverage-based or mutation-based criteria. Third, actual fault detection rates remain extremely low, often near zero, because test oracles fail to capture faulty behavior triggered by the generated test prefixes, exposing a critical limitation of automated test generation. Fourth, prompt-aware oracles can improve fault detection, but their overall effectiveness remains limited, highlighting the need for users to manually reason about test assertions. We further observe that mutation testing only marginally outperforms traditional coverage criteria in both triggering and detecting faults, raising questions about whether its significantly higher application cost is justified in this context.

Problem

• Large Language Models
• Test Adequacy Criteria
• Fault Detection

Innovation

• Large Language Models
• Test Adequacy Criteria
• Automated Test Generation
• Mutation Testing
• Prompt-aware Oracles