🤖 AI Summary
This study investigates how developers collaboratively integrate large language models (LLMs) into GitHub Pull Request (PR) workflows by sharing ChatGPT conversation links to facilitate code review and merging, specifically addressing: *What types of queries do developers pose to ChatGPT, and how do these influence their tangible contributions?*
Method: We construct the first taxonomy of 14 query categories, grounded in empirical analysis of 155 real-world PRs containing embedded ChatGPT share links; coding and qualitative analysis are performed manually.
Contribution/Results: As the first field study of human–LLM interaction within open-source development, our work systematically characterizes interaction types and collaboration patterns. Findings reveal that code review and task-oriented implementation queries dominate; code-generation requests involve significantly more conversational turns, whereas technical explanation and text polishing yield faster responses. The study elucidates the integration pathways and practical efficacy boundaries of LLMs in authentic software engineering workflows.
📝 Abstract
The emergence of Large Language Models (LLMs), such as ChatGPT, has introduced a new set of tools to support software developers in solving pro- gramming tasks. However, our understanding of the interactions (i.e., prompts) between developers and ChatGPT that result in contributions to the codebase remains limited. To explore this limitation, we conducted a manual evaluation of 155 valid ChatGPT share links extracted from 139 merged Pull Requests (PRs), revealing the interactions between developers and reviewers with ChatGPT that led to merges into the main codebase. Our results produced a catalog of 14 types of ChatGPT requests categorized into four main groups. We found a significant number of requests involving code review and the implementation of code snippets based on specific tasks. Developers also sought to clarify doubts by requesting technical explanations or by asking for text refinements for their web pages. Furthermore, we verified that prompts involving code generation generally required more interactions to produce the desired answer compared to prompts requesting text review or technical information.