Reducing Labeling Effort in Architecture Technical Debt Detection through Active Learning and Explainable AI

2026-03-03

AI Summary

This study addresses the challenge of efficiently detecting architectural technical debt (ATD), which is difficult to identify due to its high level of abstraction and strong context dependency, compounded by the prohibitive cost of manual annotation. To tackle this, the authors propose a novel approach that integrates keyword filtering, active learning, and explainable AI techniques—specifically LIME and SHAP—for ATD identification, marking the first integration of active learning with such interpretability methods in this domain. Evaluating on over 100,000 candidate issues from ten projects, the method employs query strategies such as Breaking Ties to prioritize the most informative samples for annotation. This reduces labeling effort by 49% while achieving a peak F1 score of 0.72. Expert evaluation confirms that LIME and SHAP provide clear, actionable explanations, substantially enhancing model transparency and practical utility.

Technology Category

• Machine Learning: Active Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts such as code comments, commit messages, and issue trackers. Among its types, Architecture Technical Debt (ATD) is particularly difficult to detect due to its abstract and context-dependent nature. Manual annotation of ATD is costly, time-consuming, and challenging to scale. This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI. We refined an existing dataset of 116 ATD-related Jira issues from prior work, producing 57 expert-validated items used to extract representative keywords. These were applied to identify over 103,000 candidate issues across ten open-source projects. To assess the reliability of this keyword-based filtering, we conducted a qualitative evaluation of a statistically representative sample of labeled issues. Building on this filtered dataset, we applied active learning with multiple query strategies to prioritize the most informative samples for annotation. Our results show that the Breaking Ties strategy consistently improves model performance, achieving the highest F1-score of 0.72 while reducing the annotation effort by 49\%. In order to enhance model transparency, we applied SHAP and LIME to explain the outcomes of automated ATD classification. Expert evaluation revealed that both LIME and SHAP provided reasonable explanations, with the usefulness of the explanations often depending on the relevance of the highlighted features. Notably, experts preferred LIME overall for its clarity and ease of use.

Problem

• Architecture Technical Debt
• Labeling Effort
• Active Learning
• Explainable AI
• Self-Admitted Technical Debt

Innovation

• Active Learning
• Explainable AI
• Architecture Technical Debt
• Labeling Effort Reduction
• Keyword-based Filtering