Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models

2025-07-18

AI Summary

This study systematically investigates how code context scope and prompting strategies affect the quality of unit test generation by general-purpose large language models (LLMs). We evaluate multiple mainstream models—including Gemini 2.5 Pro—under varying context configurations (e.g., function body, docstring, full implementation) and prompting techniques (e.g., chain-of-thought, CoT). Our experiments reveal that docstrings are the most critical lightweight context for improving test adequacy, whereas incorporating full implementations yields diminishing returns; notably, CoT prompting significantly enhances performance even on non-reasoning-optimized LLMs. The best configuration achieves 96.3% branch coverage, an average mutation score of 57%, and near-perfect compilation success (≈100%). To our knowledge, this is the first work to empirically quantify the efficacy of minimal context (i.e., docstrings) and generic prompting strategies for LLM-based test generation, providing both empirical validation and practical guidance for cost-effective, high-utility automated testing.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Generative AI is gaining increasing attention in software engineering, where testing remains an indispensable reliability mechanism. According to the widely adopted testing pyramid, unit tests constitute the majority of test cases and are often schematic, requiring minimal domain expertise. Automatically generating such tests under the supervision of software engineers can significantly enhance productivity during the development phase of the software lifecycle. This paper investigates the impact of code context and prompting strategies on the quality and adequacy of unit tests generated by various large language models (LLMs) across several families. The results show that including docstrings notably improves code adequacy, while further extending context to the full implementation yields definitely smaller gains. Notably, the chain-of-thought prompting strategy -- applied even to 'reasoning' models -- achieves the best results, with up to 96.3% branch coverage, a 57% average mutation score, and near-perfect compilation success rate. Among the evaluated models, M5 (Gemini 2.5 Pro) demonstrated superior performance in both mutation score and branch coverage being still in top in terms of compilation success rate. All the code and resulting test suites are publicly available at https://github.com/peetery/LLM-analysis.

Problem

• Impact of code context on unit test generation with LLMs
• Effect of prompting strategies on test quality and adequacy
• Comparison of LLM performance in automated test generation

Innovation

• Utilizes docstrings to enhance code adequacy
• Applies chain-of-thought prompting strategy
• Evaluates multiple large language models