Assessing Language Models for Salient Class Identification

2026-06-19

AI Summary

This work addresses the challenge of identifying the dominant change intent among multiple modified classes in code commits. It proposes a novel approach that leverages large language models to directly infer the most significant class from raw source code, eliminating the need for abstract syntax tree (AST) parsing, dependency graphs, or handcrafted features. The study presents the first systematic evaluation of various open- and closed-source models—including GPT-5.4 and Qwen3.5-9B—under zero-shot, few-shot, and chain-of-thought prompting settings. Results demonstrate that language models substantially outperform the strongest existing baselines, with lightweight open-source models such as Qwen3.5-9B achieving performance close to GPT-5.4 in few-shot scenarios, thereby confirming the practical feasibility of deploying compact, locally hosted models in industrial applications.

Abstract

Code review requires reviewers to understand the core intent of code changes, which becomes difficult when a commit modifies multiple classes. In such commits, one or more primarily modified classes, referred to as salient classes, may induce modifications in other classes. Accurate identification of salient classes offers reviewers an effective entry point to navigate code changes and facilitates program comprehension. Existing state-of-the-art approaches rely on complex program-analysis procedures, including Abstract Syntax Tree (AST) parsing, class relation extraction, handcrafted feature engineering, or dependency graph construction. To this end, we study whether language models (LMs) can identify salient classes directly from commits without feature engineering, graph construction, or training. We first construct a new dataset ApacheJavaCM, derived from the ApacheCM dataset, containing 7,911 commits and 25,914 labeled classes. On this dataset, we systematically evaluate whether LMs can identify salient classes directly from commits and compare with the strongest reproducible state-of-the-art (SOTA) baseline. The evaluation covers two large language models (LLMs), GPT-5.4 and DeepSeek-V3.2, one small language model (SLM), Qwen3.5-9B, and three prompting strategies: zero-shot, few-shot, and chain-of-thought. The LMs substantially outperform the baseline while remaining stable across commit characteristics and selected LMs. We also found that, for salient class identification tasks, a 9B-parameter open-source SLM, Qwen3.5-9B, under few-shot prompting, achieves performance comparable to that of a much larger closed-source LLM, GPT-5.4. These results suggest that lightweight, locally deployable SLMs are sufficient for industrial salient class identification tasks and can reduce both cost and privacy barriers associated with relying on closed-source LLMs.

Problem

• salient class identification
• code review
• commit analysis
• program comprehension
• language models

Innovation

• salient class identification
• language models
• code review
• prompting strategies
• open-source SLMs