SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

2026-05-21

AI Summary

Current large language models (LLMs) often generate superficial and insufficiently discriminative test suites, which fail to provide precise feedback for program repair and reinforcement learning. To address this limitation, this work proposes SWE-Mutation—the first mutation-based benchmark for evaluating test suite effectiveness—featuring a language-agnostic agent framework that systematically produces high-fidelity, multi-language, complex mutant programs. The benchmark spans nine programming languages and comprises 2,636 mutation instances. Experiments on seven prominent LLMs reveal that even the best-performing model, DeepSeek-V3.1, achieves only a 10.20% validation rate and a 36.15% detection rate. Moreover, the newly introduced mutation strategies reduce the average detection rate from 71.04% to 39.81%, exposing significant deficiencies in the current LLMs’ ability to generate effective tests.

Abstract

Evaluating software engineering capabilities has become a core component of modern large language models (LLMs); however, the key bottleneck hindering further scaling lies not in the scarcity of high-quality solutions, but in the lack of high-quality test suites. Test suites are indispensable both for synthesizing program repair trajectories and for providing precise feedback signals in reinforcement learning. Unfortunately, due to the high cost and difficulty of annotation, high-quality test suites have long been hard to obtain, while those automatically generated by LLMs tend to be superficial and lack sufficient discriminative power. As a first step toward constructing high-quality test suites, we introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites. The benchmark characterizes test suites by introducing systematically mutated solutions that attempt to ``fool'' the test suites and pass validation. We further propose an agentic, language-agnostic framework for automatically generating complex mutants. Our benchmark consists of 2,636 mutated variants derived from 800 original instances and includes a multilingual subset spanning nine programming languages. Experiments on seven LLMs reveal that even DeepSeek-V3.1 achieves only 10.20% verification and 36.15% detection rates, highlighting the inadequacy of current LLMs. Additionally, our agentic mutation strategy enhances realism, reducing average detection rates from 71.04% to 39.81% compared to conventional methods. These findings expose persistent deficiencies in the ability of current LLMs to generate reliable and discriminative test suites.

Problem

• test suite generation
• large language models
• software engineering
• mutation testing
• discriminative power

Innovation

• SWE-Mutation
• test suite generation
• code mutation
• large language models
• software engineering evaluation