Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

2026-02-08

AI Summary

This study critically examines and empirically evaluates the practical utility of large language model (LLM) agents in automatically generating test cases for autonomous software engineering. Leveraging the SWE-bench Verified benchmark, we analyze execution traces from six state-of-the-art LLM agents, modulating test generation frequency through prompt engineering and conducting controlled experiments with statistical assessment to systematically investigate the contribution of testing behavior to bug repair. Our findings reveal no significant correlation between test generation frequency and task success rate. Moreover, agents predominantly rely on print statements rather than assertions for verification, suggesting that current auto-generated tests primarily serve as observational feedback rather than enabling effective self-repair. This work presents the first systematic characterization of the effectiveness boundaries of LLM agents’ testing practices in autonomous software repair.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Teamwork
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, a paradigm adopted by many high-ranking agents on the SWE-bench leaderboard. However, we observe that GPT-5.2, which writes almost no new tests, can even achieve performance comparable to top-ranking agents. This raises the critical question: whether such tests meaningfully improve issue resolution or merely mimic human testing practices while consuming a substantial interaction budget. To reveal the impact of agent-written tests, we present an empirical study that analyzes agent trajectories across six state-of-the-art LLMs on SWE-bench Verified. Our results show that while test writing is commonly adopted, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies Furthermore, these tests typically serve as observational feedback channels, where agents prefer value-revealing print statements significantly more than formal assertion-based checks. Based on these insights, we perform a controlled experiment by revising the prompts of four agents to either increase or reduce test writing. The results suggest that changes in the volume of agent-written tests do not significantly change final outcomes. Taken together, our study reveals that current test-writing practices may provide marginal utility in autonomous software engineering tasks.

Problem

• LLM-based software engineering agents
• agent-generated tests
• test utility
• autonomous software engineering
• SWE-bench

Innovation

• LLM-based software engineering agents
• agent-generated tests
• SWE-bench
• empirical study
• test utility