Automated Test Generation from Program Documentation Encoded in Code Comments

2025-04-29

AI Summary

Coverage-driven testing often fails to capture semantic behavior and misses deep-seated defects. To address this, this paper proposes a behavior-oriented test generation method leveraging code comments (e.g., Javadoc). First, functional specifications are extracted from natural-language comments and formally modeled as executable test objectives. Second, search-based test generation is integrated with context-aware assertion synthesis to automatically generate named, behaviorally meaningful test cases. This work is the first to explicitly transform code comments into executable test goals, thereby overcoming the semantic blindness of conventional coverage metrics and enabling precise, behavior-level testing. Evaluated on a benchmark of 118 Java classes, our approach significantly improves behavioral coverage and successfully detects multiple previously unknown defects that evade state-of-the-art coverage-based tools.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Documenting the functionality of software units with code comments, e.g., Javadoc comments, is a common programmer best-practice in software engineering. This paper introduces a novel test generation technique that exploits the code-comment documentation constructively. We originally address those behaviors as test objectives, which we pursue in search-based fashion. We deliver test cases with names and oracles properly contextualized on the target behaviors. Our experiments against a benchmark of 118 Java classes indicate that the proposed approach successfully tests many software behaviors that may remain untested with coverage-driven test generation approaches, and distinctively detects unknown failures.

Problem

• Generates tests from code comments automatically
• Targets untested behaviors in coverage-driven approaches
• Detects unknown failures via contextualized test cases

Innovation

• Generates tests from code comments
• Uses search-based test objectives
• Produces contextualized test cases