Towards Refining Developer Questions using LLM-Based Named Entity Recognition for Developer Chatroom Conversations

2025-03-02

AI Summary

Ambiguous problem formulations in developer chatrooms hinder effective resolution. Method: This paper proposes SENIR, a multi-task framework for software dialogues that jointly models named entity recognition (NER), user intent, and problem solvability status. SENIR introduces the first large language model (LLM)-based, domain-specific multi-task annotation paradigm for software communication, unifying the identification of technical entities, user intents, and resolution states. It further reveals intent-dependent variations in key influencing factors. Evaluation employs SMOTE oversampling, cross-validation, and bootstrapping. Results: On the DISCO dataset, SENIR achieves 86% F1 for NER, 71% accuracy for intent detection, 89% F1 for solvability classification, and 0.7–0.8 AUC for resolution prediction. Analysis identifies programming language, user-defined variables, and positive sentiment as statistically significant facilitators of problem resolution.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Multiagent Systems: Distributed Problem Solving

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In software engineering chatrooms, communication is often hindered by imprecise questions that cannot be answered. Recognizing key entities can be essential for improving question clarity and facilitating better exchange. However, existing research using natural language processing techniques often overlooks these software-specific nuances. In this paper, we introduce Software-specific Named Entity Recognition, Intent Detection, and Resolution Classification (SENIR), a labeling approach that leverages a Large Language Model to annotate entities, intents, and resolution status in developer chatroom conversations. To offer quantitative guidance for improving question clarity and resolvability, we build a resolution prediction model that leverages SENIR's entity and intent labels along with additional predictive features. We evaluate SENIR on the DISCO dataset using a subset of annotated chatroom dialogues. SENIR achieves an 86% F-score for entity recognition, a 71% F-score for intent detection, and an 89% F-score for resolution status classification. Furthermore, our resolution prediction model, tested with various sampling strategies (random undersampling and oversampling with SMOTE) and evaluation methods (5-fold cross-validation, 10-fold cross-validation, and bootstrapping), demonstrates AUC values ranging from 0.7 to 0.8. Key factors influencing resolution include positive sentiment and entities such as Programming Language and User Variable across multiple intents, while diagnostic entities are more relevant in error-related questions. Moreover, resolution rates vary significantly by intent: questions about API Usage and API Change achieve higher resolution rates, whereas Discrepancy and Review have lower resolution rates. A Chi-Square analysis confirms the statistical significance of these differences.

Problem

• Improves question clarity in developer chatrooms using LLM-based entity recognition.
• Develops SENIR for labeling entities, intents, and resolution status in chat conversations.
• Predicts question resolution rates using entity and intent labels with additional features.

Innovation

• LLM-based Named Entity Recognition for chatrooms
• Resolution prediction model with SENIR labels
• Evaluation using DISCO dataset and cross-validation