Automated Classification of Human Code Review Comments with Large Language Models

2026-04-26

AI Summary

This study addresses common issues in human-written code review comments—such as redundancy, ambiguity, and lack of constructiveness—and the absence of a fine-grained classification scheme. The authors propose a unified nine-label taxonomy that integrates six types of comment defects with three categories of useful intent. Using manually annotated data, they systematically evaluate zero-shot and one-shot prompting strategies with large language models—including GPT-5-mini, LLaMA-3.3, and DeepSeek-R1—taking as input both the review comment text and its associated unified diff code snippet. Experimental results show that zero-shot classification achieves macro F1 scores between 0.360 and 0.374. One-shot prompting significantly improves performance for GPT-5-mini and DeepSeek-R1, particularly on boundary intent classes, though it leads to a slight decline for LLaMA-3.3.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Context: Code reviews are essential for maintaining software quality, yet many human review comments suffer from issues such as redundancy, vagueness, or lack of constructiveness. These types of comments may slow down feedback and obscure important insights. Prior work on code review comments mostly explore the detection and categorization of useful comments, while fine-grained categorization of comment issues remains underexplored. Objective: This work aims to design and evaluate an automated system for classifying code review comments according to specific categories of issues. Methodology: We introduced a nine-label taxonomy for code review comments, covering six review comment smells and three common useful intents, and manually labeled 448 comments from a publicly available dataset. We benchmarked zero-shot and one-shot single-label classification over each comment and its associated unified diff hunk, comparing GPT-5-mini, LLaMA-3.3, and DeepSeek-R1. We reported macro-F1 as the primary metric. Results: Zero-shot performance was moderate under class imbalance (macro-F1 0.360 to 0.374). One-shot exemplar conditioning had model-dependent effects: GPT-5-mini and DeepSeek-R1 macro-F1 scores improved, however LLaMA-3.3 suffered a slight decrease. Exemplars most consistently helped intent-boundary labels, whereas classification of evidence-sensitive labels remain challenging. Conclusion: Our results indicate that comment--diff evidence is sufficient for some labels but limited for evidence-sensitive smells. Future work includes adding thread context, improving intent-preserving rewrites, and validating robustness across platforms.

Problem

• code review comments
• comment classification
• review comment smells
• constructiveness
• redundancy

Innovation

• code review comment classification
• comment smells
• large language models
• few-shot learning
• diff-aware classification