Issue Retrieval and Verification Enhanced Supplementary Code Comment Generation

2025-06-17

AI Summary

To address the pervasive hallucination problem in retrieval-augmented code comment generation, this paper proposes the first issue-oriented three-stage framework: retrieval–generation–verification. We systematically categorize five types of design rationale information extractable from software issue reports and develop a multi-source semantic retrieval mechanism integrating code, comments, and issues. Furthermore, we introduce a logic-based verifiability discriminator grounded in issue evidence to explicitly filter out unverifiable or code-irrelevant hallucinated comments. Experiments demonstrate substantial improvements in human-annotated comment coverage: +38.6% (72.2%) for ChatGPT, +52.6% (88.4%) for GPT-4o, and +51.2% (86.2%) for DeepSeek-V3. The proposed method also achieves consistent superiority across all MESIA metrics, validating its dual advancement in reliability and practical utility for industrial-grade comment generation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Issue reports have been recognized to contain rich information for retrieval-augmented code comment generation. However, how to minimize hallucinations in the generated comments remains significant challenges. In this paper, we propose IsComment, an issue-based LLM retrieval and verification approach for generating method's design rationale, usage directives, and so on as supplementary code comments. We first identify five main types of code supplementary information that issue reports can provide through code-comment-issue analysis. Next, we retrieve issue sentences containing these types of supplementary information and generate candidate code comments. To reduce hallucinations, we filter out those candidate comments that are irrelevant to the code or unverifiable by the issue report, making the code comment generation results more reliable. Our experiments indicate that compared with LLMs, IsComment increases the coverage of manual supplementary comments from 33.6% to 72.2% for ChatGPT, from 35.8% to 88.4% for GPT-4o, and from 35.0% to 86.2% for DeepSeek-V3. Compared with existing work, IsComment can generate richer and more useful supplementary code comments for programming understanding, which is quantitatively evaluated through the MESIA metric on both methods with and without manual code comments.

Problem

• Minimizing hallucinations in generated code comments
• Retrieving issue reports for supplementary comment generation
• Verifying relevance of comments to code and issue reports

Innovation

• Issue-based LLM retrieval for comment generation
• Filtering unverifiable comments to reduce hallucinations
• Enhancing coverage and reliability of supplementary comments