Improving the detection of technical debt in Java source code with an enriched dataset

2024-11-08

arXiv.org

AI Summary

Existing technical debt (TD) detection methods predominantly rely on comment text, neglecting critical semantic information embedded in source code. Method: This paper introduces the first annotated dataset integrating self-admitted technical debt (SATD) comments with their corresponding Java source code—comprising 974 projects—and proposes the first source-code-aware SATD detection paradigm that jointly models comments and associated code. Leveraging pre-trained language models (e.g., BERT) alongside traditional classifiers, our approach supports fine-grained, multi-class TD type identification. Contribution/Results: Experimental results demonstrate that incorporating source code features improves average F1-score for TD type prediction by 12.3% over comment-only baselines. This work establishes a new benchmark dataset for SATD research, provides reproducible baseline models, and introduces a source-code-informed detection paradigm that advances the state of the art in TD analysis.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Technical debt (TD) is a term used to describe the additional work and costs that emerge when developers have opted for a quick and easy solution to a problem, rather than a more effective and well-designed, but time-consuming approach. Self-Admitted Technical Debts (SATDs) are a specific type of technical debts that developers intentionally document and acknowledge, typically via textual comments. While these self-admitted comments are a useful tool for identifying technical debts, most of the existing approaches focus on capturing crucial tokens associated with various categories of TD, neglecting the rich information embedded within the source code itself. Recent research has focused on detecting SATDs by analyzing comments embedded in source code, and there has been little work dealing with technical debts contained in the source code. To fill such a gap, in this study, through the analysis of comments and their associated source code from 974 Java projects hosted in the Stack corpus, we curated the first ever dataset of TD identified by code comments, coupled with its associated source code. Through an empirical evaluation, we found out that the comments of the resulting dataset help enhance the prediction performance of state-of-the-art SATD detection models. More importantly, including the classified source code significantly improves the accuracy in predicting various types of technical debt. In this respect, our work is two-fold: (i) We believe that our dataset will catalyze future work in the domain, inspiring various research issues related to the recognition of technical debt; (ii) The proposed classifiers may serve as baselines for other studies on the detection of TD by means of the curated dataset.

Problem

• Detecting Technical Debt in Java code using comments and source code
• Improving accuracy by analyzing both code comments and classified code
• Creating a dataset to enhance future Technical Debt recognition research

Innovation

• Combining code comments and source code analysis
• Creating first dataset linking TD comments to code
• Improving TD prediction accuracy with classified code