How I Learned to Stop Worrying and Love ChatGPT

2024-04-15

IEEE Working Conference on Mining Software Repositories

AI Summary

This study investigates the evolutionary patterns and long-term persistence of ChatGPT-generated code in real-world software projects. Addressing concerns about AI-generated code’s impact on development workflows and ecosystem sustainability, we introduce survival analysis—specifically the Cox proportional hazards model—to this domain for the first time. We propose a fine-grained code–dialogue alignment technique and context attribution method to disentangle contributions from user inputs versus model outputs. Leveraging large-scale code change logs, static analysis, and session-level dialogue tracing, our quantitative analysis reveals that approximately 38% of ChatGPT-related code is modified or deleted within six months—significantly exceeding the attrition rate of human-written code. In contrast, code generated from high-context-fidelity dialogues exhibits a 2.3× higher survival rate. This work establishes a reproducible methodological framework and empirical benchmark for rigorously assessing the practical impact of AI programming assistants.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Other Foundations of Human Computation & AI
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In the dynamic landscape of software engineering, the emergence of ChatGPT-generated code signifies a distinctive and evolving paradigm in development practices. We delve into the impact of interactions with ChatGPT on the software development process, specifically analysing its influence on source code changes. Our emphasis lies in aligning code with ChatGPT conversations, separately analysing the user-provided context of the code and the extent to which the resulting code has been influenced by ChatGPT. Additionally, employing survival analysis techniques, we examine the longevity of ChatGPT-generated code segments in comparison to lines written traditionally. The goal is to provide valuable insights into the transformative role of ChatGPT in software development, illuminating its implications for code evolution and sustainability within the ecosystem.CCS CONCEPTS• Software and its engineering → Software prototyping; Software evolution; Automatic programming.

Problem

• Impact of ChatGPT interactions on software development process
• Alignment of code with ChatGPT conversations and context
• Longevity comparison of ChatGPT-generated vs traditional code

Innovation

• Aligning code with ChatGPT conversations
• Analyzing user context and ChatGPT influence
• Comparing longevity of ChatGPT vs traditional code