ReFEree: Reference-Free and Fine-Grained Method for Evaluating Factual Consistency in Real-World Code Summarization

2026-04-12

AI Summary

Existing approaches struggle to perform fine-grained factual consistency evaluation for code summaries in real-world scenarios, particularly when summaries span multiple sentences and rely on contextual dependencies. This work proposes the first reference-free, fine-grained evaluation method tailored to code summarization: it defines domain-specific criteria for factual inconsistency, analyzes summaries at the segment level while incorporating dependency-aware context, and aggregates rule-based matches into an overall consistency score. To support this approach, we introduce the first benchmark dataset with human-annotated, fine-grained factual consistency labels. Experimental results demonstrate that our method significantly outperforms 13 baselines in correlation with human judgments, achieving a 15–18% improvement over the current state-of-the-art.

Technology Category

• Natural Language Processing: Summarization
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As Large Language Models (LLMs) have become capable of generating long and descriptive code summaries, accurate and reliable evaluation of factual consistency has become a critical challenge. However, previous evaluation methods are primarily designed for short summaries of isolated code snippets. Consequently, they struggle to provide fine-grained evaluation of multi-sentence functionalities and fail to accurately assess dependency context commonly found in real-world code summaries. To address this, we propose ReFEree, a reference-free and fine-grained method for evaluating factual consistency in real-world code summaries. We define factual inconsistency criteria specific to code summaries and evaluate them at the segment level using these criteria along with dependency information. These segment-level results are then aggregated into a fine-grained score. We construct a code summarization benchmark with human-annotated factual consistency labels. The evaluation results demonstrate that ReFEree achieves the highest correlation with human judgment among 13 baselines, improving 15-18% over the previous state-of-the-art. Our code and data are available at https://github.com/bsy99615/ReFEree.git.

Problem

• factual consistency
• code summarization
• reference-free evaluation
• fine-grained evaluation
• real-world code

Innovation

• factual consistency
• code summarization
• reference-free evaluation
• fine-grained assessment
• dependency context