ASTAGEN: Empirical Evaluation of Automated SATD Taxonomy Generation with LLMs

2025-06-11

AI Summary

SATD taxonomy construction is time-consuming, highly subjective, and suffers from poor inter-annotator consistency. Method: This paper proposes an LLM-based automated SATD classification method featuring an explanation-driven iterative prompting framework that supports dynamic category generation, evolution, and incremental clustering, integrated with context-aware joint modeling of code and comments. Contribution/Results: The approach is the first to empirically validate SATD auto-classification feasibility and generalizability across quantum software, smart contracts, and machine learning domains. Experiments show it significantly outperforms baseline direct-LLM approaches in reproducing prior categories (e.g., “Layer Configuration” in ML), with maximum dataset processing time under two hours and cost below one dollar. Classification consistency improves markedly, enabling efficient, low-cost, semi-automated construction of domain-specific SATD taxonomies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. This study presents ASTAGEN, an initial step toward automating SATD taxonomy generation using large language models (LLMs). Given a comment and its surrounding code, ASTAGEN first generates a concise explanation for each SATD comment, then incrementally generates and updates categories to construct a taxonomy. We evaluate ASTAGEN on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovers domain-specific categories reported in prior work, such as Layer Configuration in machine learning. Compared to a naive use of an LLM, ASTAGEN produces more consistent category assignments due to its explanation-driven, iterative design. It also completes taxonomy generation in under two hours and for less than one USD, even on the largest dataset. These results suggest that while full automation remains challenging, ASTAGEN is able to support semi-automated taxonomy construction. Furthermore, our work opens up avenues for future work, such as automatic taxonomy generation in other areas.

Problem

• Automating SATD taxonomy generation to reduce manual effort
• Improving consistency in SATD category assignments using LLMs
• Enabling cost-effective taxonomy construction across diverse software domains

Innovation

• LLM-based automated SATD taxonomy generation
• Explanation-driven iterative category construction
• Cost-efficient domain-specific taxonomy recovery