Evaluating the Performance and Efficiency of Sentence-BERT for Code Comment Classification

2025-06-10

AI Summary

This work systematically evaluates the accuracy-efficiency trade-off of Sentence-BERT in multi-label code comment classification. Using a manually annotated dataset of 13,216 statements, we fine-tune Sentence-BERT and design multiple lightweight multi-label classification heads to maximize F1-score while preserving inference feasibility. We conduct the first quantitative analysis of model size impact on latency (+1.4×) and computational cost (GFLOPS +2.1×), and propose a balanced deployment strategy that improves F1 by 0.0346 without increasing inference overhead. Our core contribution lies in uncovering the co-optimization principle between semantic encoders and classification heads in code-domain multi-label classification, and establishing a reproducible, lightweight deployment paradigm that jointly optimizes encoder capacity and head architecture for practical code understanding systems.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This work evaluates Sentence-BERT for a multi-label code comment classification task seeking to maximize the classification performance while controlling efficiency constraints during inference. Using a dataset of 13,216 labeled comment sentences, Sentence-BERT models are fine-tuned and combined with different classification heads to recognize comment types. While larger models outperform smaller ones in terms of F1, the latter offer outstanding efficiency, both in runtime and GFLOPS. As result, a balance between a reasonable F1 improvement (+0.0346) and a minimal efficiency degradation (+1.4x in runtime and +2.1x in GFLOPS) is reached.

Problem

• Evaluate Sentence-BERT for code comment classification
• Balance performance and efficiency in classification
• Compare model sizes for F1 and resource usage

Innovation

• Fine-tuned Sentence-BERT for comment classification
• Combined with diverse classification heads
• Balanced performance and efficiency trade-offs