Do Automatic Comment Generation Techniques Fall Short? Exploring the Influence of Method Dependencies on Code Understanding

2025-04-28

AI Summary

Existing method-level comment generation techniques ignore inter-method call dependencies, causing a substantial degradation in comment quality—approximately 70% of Java methods with such dependencies exhibit markedly poorer annotations. This work is the first to systematically characterize the prevalence and adverse impact of method dependencies on comment generation. We propose HelpCOM, a dependency-aware framework that integrates static call graph analysis, context-enhanced code encoding, and multi-stage prompt engineering to explicitly incorporate information from invoked helper methods, thereby improving comment clarity, completeness, and relevance. HelpCOM is compatible with mainstream models including CodeT5+, CodeBERT, and ASAP. Evaluated on 448K dependent methods, it achieves improvements of 5.6%–50.4% over state-of-the-art approaches across BLEU, Sentence-BERT, and LLM-based metrics. A developer study involving 156 participants confirms that HelpCOM significantly enhances code understandability and maintenance efficiency.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Metareasoning and Metaheuristics
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Method-level comments are critical for improving code comprehension and supporting software maintenance. With advancements in large language models (LLMs), automated comment generation has become a major research focus. However, existing approaches often overlook method dependencies, where one method relies on or calls others, affecting comment quality and code understandability. This study investigates the prevalence and impact of dependent methods in software projects and introduces a dependency-aware approach for method-level comment generation. Analyzing a dataset of 10 popular Java GitHub projects, we found that dependent methods account for 69.25% of all methods and exhibit higher engagement and change proneness compared to independent methods. Across 448K dependent and 199K independent methods, we observed that state-of-the-art fine-tuned models (e.g., CodeT5+, CodeBERT) struggle to generate comprehensive comments for dependent methods, a trend also reflected in LLM-based approaches like ASAP. To address this, we propose HelpCOM, a novel dependency-aware technique that incorporates helper method information to improve comment clarity, comprehensiveness, and relevance. Experiments show that HelpCOM outperforms baseline methods by 5.6% to 50.4% across syntactic (e.g., BLEU), semantic (e.g., SentenceBERT), and LLM-based evaluation metrics. A survey of 156 software practitioners further confirms that HelpCOM significantly improves the comprehensibility of code involving dependent methods, highlighting its potential to enhance documentation, maintainability, and developer productivity in large-scale systems.

Problem

• Investigates impact of method dependencies on comment quality
• Proposes dependency-aware technique for better comment generation
• Improves code comprehension for dependent methods significantly

Innovation

• Dependency-aware method for comment generation
• Incorporates helper method information
• Outperforms baselines in multiple metrics