CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement

2025-05-26

AI Summary

Existing code comment quality evaluation methods (e.g., SIDE, MIDQ, STASIS) rely on reference texts, language-specific assumptions, or single-dimensional metrics, limiting their utility for high-quality dataset construction. To address this, we propose CIDRe—a language-agnostic, reference-free, multidimensional evaluation framework that quantifies comment quality along four synergistic dimensions: relevance, informativeness, completeness, and descriptive length. CIDRe is the first to integrate four parameter-free metrics: semantic alignment (via embedding-based similarity), functional coverage (via template matching), structural completeness (via information-entropy-driven coverage estimation), and detail sufficiency (via length-adaptive normalization). Evaluated on human-annotated benchmarks, CIDRe significantly outperforms baseline methods—achieving substantial gains in cross-entropy assessment—and GPT-4o-mini–based human evaluation confirms its ability to reliably detect quality improvements in fine-tuned model outputs (p < 0.01).

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Effective generation of structured code comments requires robust quality metrics for dataset curation, yet existing approaches (SIDE, MIDQ, STASIS) suffer from limited code-comment analysis. We propose CIDRe, a language-agnostic reference-free quality criterion combining four synergistic aspects: (1) relevance (code-comment semantic alignment), (2) informativeness (functional coverage), (3) completeness (presence of all structure sections), and (4) description length (detail sufficiency). We validate our criterion on a manually annotated dataset. Experiments demonstrate CIDRe's superiority over existing metrics, achieving improvement in cross-entropy evaluation. When applied to filter comments, the models finetuned on CIDRe-filtered data show statistically significant quality gains in GPT-4o-mini assessments.

Problem

• Lack robust metrics for code comment quality measurement
• Existing approaches analyze code-comment limitedly
• Need language-agnostic reference-free quality criterion

Innovation

• Language-agnostic reference-free quality criterion
• Combines four synergistic code-comment aspects
• Validated on manually annotated dataset