ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts

2025-09-09

AI Summary

This study investigates how developers articulate code refactoring requirements during interactions with ChatGPT and how large language models (LLMs) recognize and respond to such intents. Using a corpus of 29,778 real-world developer–ChatGPT dialogues, we applied text mining and qualitative analysis to extract and annotate 715 refactoring-related utterances—enabling the first systematic identification and taxonomy of *explicit* refactoring intents. We characterize prevalent refactoring topics (e.g., readability, performance, maintainability) and identify high-efficacy prompting patterns, notably structured descriptions augmented with contextual code snippets and goal-oriented phrasing. The findings provide empirical grounding for intent modeling in human–model collaborative refactoring. Moreover, the distilled prompt features directly inform the design and optimization of AI-powered development tools, including IDE plugins and programming assistants.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs), such as ChatGPT, have become widely popular and widely used in various software engineering tasks such as refactoring, testing, code review, and program comprehension. Although recent studies have examined the effectiveness of LLMs in recommending and suggesting refactoring, there is a limited understanding of how developers express their refactoring needs when interacting with ChatGPT. In this paper, our goal is to explore interactions related to refactoring between developers and ChatGPT to better understand how developers identify areas for improvement in code, and how ChatGPT addresses developers' needs. Our approach involves text mining 715 refactoring-related interactions from 29,778 ChatGPT prompts and responses, as well as the analysis of developers' explicit refactoring intentions.

Problem

• Analyzing developer-ChatGPT interactions for refactoring needs
• Understanding how developers express code improvement requirements
• Evaluating ChatGPT's effectiveness in addressing refactoring prompts

Innovation

• Text mining refactoring interactions from ChatGPT
• Analyzing explicit developer refactoring intentions
• Examining prompt-response patterns for code improvement