BuggIn: Automatic Intrinsic Bugs Classification Model using NLP and ML

2023-12-08

International Conference on Predictive Models in Software Engineering

AI Summary

This paper addresses the lack of automated methods for identifying intrinsic bugs in software defect classification. We propose, for the first time, a purely text-driven end-to-end framework for automatic intrinsic bug identification. Our approach fuses semantic embeddings from seBERT with TF-IDF features and employs classical machine learning models—including SVM, logistic regression, and decision trees—to classify root-cause types directly from defect report titles and descriptions, eliminating the need for manual labeling based on version control histories. Experimental results demonstrate that TF-IDF combined with a decision tree achieves an F1-score of 78% on title text, while seBERT with SVM attains 77%. These results robustly validate the effectiveness and practicality of identifying intrinsic bugs using textual information alone, thereby overcoming the longstanding bottleneck of relying on code change history for annotation.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Semi-Supervised Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent studies have shown that bugs can be categorized into in- trinsic and extrinsic types. Intrinsic bugs can be backtracked to specific changes in the version control system (VCS), while extrin- sic bugs originate from external changes to the VCS and lack a direct bug-inducing change. Using only intrinsic bugs to train bug prediction models has been reported as beneficial to improve the performance of such models. However, there is currently no auto- mated approach to identify intrinsic bugs. To bridge this gap, our study employs Natural Language Processing (NLP) techniques to automatically identify intrinsic bugs. Specifically, we utilize two embedding techniques, seBERT and TF-IDF, applied to the title and description text of bug reports. The resulting embeddings are fed into well-established machine learning algorithms such as Support Vector Machine, Logistic Regression, Decision Tree, Random Forest, and K-Nearest Neighbors. The primary objective of this paper is to assess the performance of various NLP and machine learning techniques in identifying intrinsic bugs using the textual informa- tion extracted from bug reports. The results demonstrate that both seBERT and TF-IDF can be effectively utilized for intrinsic bug identification. The highest performance scores were achieved by combining TF-IDF with the Decision Tree algorithm and utilizing the bug titles (yielding an F1 score of 78%). This was closely fol- lowed by seBERT, Support Vector Machine, and bug titles (with an F1 score of 77%). In summary, this paper introduces an innovative approach that automates the identification of intrinsic bugs using textual information derived from bug reports.

Problem

• Automatically classify intrinsic bugs using NLP and ML
• Identify intrinsic bugs from bug reports' textual information
• Evaluate NLP and ML techniques for bug classification performance

Innovation

• NLP techniques for intrinsic bug identification
• seBERT and TF-IDF embeddings for bug reports
• Machine learning algorithms classify intrinsic bugs