Humanizing Automatically Generated Unit Test Suites with LLM-Based Refactoring

2026-06-26

AI Summary

This work addresses the limitations of existing automated unit test generation: while search-based software testing (SBST) tools like EvoSuite produce highly covering yet poorly readable tests, directly generating tests with large language models (LLMs) often yields low compilation success rates. To bridge this gap, the authors propose TestHumanizer, which innovatively employs an LLM as a controlled refactoring layer to enhance SBST-generated, compilable test suites. The approach significantly improves test naming, structure, and readability while preserving behavioral semantics and maintaining high code coverage—sacrificing only 1–2 percentage points. Evaluated on 350 classes yielding 31,500 tests, TestHumanizer achieves compilation rates of 88–98% and meets fidelity criteria in 86–95% of cases. User studies confirm substantially improved developer-perceived readability and willingness to adopt the generated tests (p<0.01).

Abstract

Search-based test generation tools such as EvoSuite produce compilable and high-coverage unit tests at scale, but their suites are often hard to read and maintain. LLMs can generate more natural tests, yet direct generation remains brittle, with compilation rates of only 51-78% in our study. We introduce TestHumanizer, a hybrid SBST+LLM approach that uses LLMs as controlled refactoring layers over compilable SBST suites to improve naming, structure, and developer-oriented clarity while preserving behavior and compilation validity. We evaluate TestHumanizer on 350 classes from Defects4J and SF110. EvoSuite generates 15 suites per class, and each suite is refactored under three context configurations using gpt-4o and mistral-large-2407, yielding 31,500 refactorings. TestHumanizer reaches 88-98% compilation rates, close to EvoSuite's 100% baseline and clearly above direct LLM generation. Structural coverage is largely preserved, typically within 1-2 percentage points, and 86-95% of refactorings satisfy a composite faithful-refactoring threshold. Refactored suites also improve predicted readability, reduce control-flow and cognitive complexity, and mitigate structural smells. The summary-based setting offers the most robust trade-off, while long code-centric prompts are more prone to hallucination-induced failures. A developer study on 30 classes and 444 test methods confirms significant gains in perceived readability and willingness to adopt, with Wilcoxon p less than 0.01 and substantial inter-rater agreement. Overall, LLMs are most effective not as standalone generators but as validation-gated refinement layers over robust SBST outputs.

Problem

• unit test generation
• test readability
• LLM-based refactoring
• search-based software testing
• test maintainability

Innovation

• LLM-based refactoring
• search-based software testing
• test suite humanization
• compilation-preserving refinement
• developer-oriented test readability