Problem
Research questions and friction points this paper is trying to address.
AI-assisted programming
conversational programming
IDE-integrated AI
empirical study
developer behavior
Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
🤖 AI Summary
This study addresses a critical gap in the empirical understanding of conversational AI programming behaviors within real-world development environments, as prior work has largely relied on small-scale experiments or generic chatbots. Analyzing 11,579 authentic IDE sessions comprising 74,998 developer messages through large-scale log analysis, conversation coding, and qualitative induction, this work reveals three paradigm shifts in how developers collaborate with AI in codebase-aware contexts: incremental requirement articulation, offloading of cognitive tasks to AI, and proactive collaboration management. The findings demonstrate that developers achieve effective coordination by iteratively refining prompts, injecting contextual information, and imposing constraints—strategies that offer crucial empirical grounding and design implications for next-generation AI-powered programming tools.
📝 Abstract
IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.
Innovation
Methods, ideas, or system contributions that make the work stand out.
conversational programming
AI-assisted IDE
large-scale behavioral analysis
codebase-aware AI
developer-AI collaboration
🔎 Similar Papers
No similar papers found.
