Prompt Quality and Pull Request Outcomes: A Stage-Based Empirical Study of LLM-Assisted Development

2026-06-17

AI Summary

This study investigates how prompt structure influences downstream collaborative integration behaviors in pull requests (PRs) during large language model (LLM)-assisted software development. Building on three dimensions—context, specificity, and verifiability—the authors develop an analytical framework for prompt structure and introduce a hybrid human–LLM annotation approach to address inconsistencies in LLM-based prompt evaluation. Empirical analysis reveals that specificity most effectively facilitates the generation of executable code, verifiability predominantly governs code adoption decisions, and context significantly affects integration depth. This work establishes, for the first time, a phased linkage between prompt structure and PR outcomes, uncovering the differential roles of prompt characteristics across distinct stages of the development process.

Abstract

Large language model (LLM)-powered tools such as ChatGPT are increasingly used in collaborative software engineering workflows, yet little is known about how prompt structure influences downstream pull request (PR) outcomes. Prior studies primarily examine conversational helpfulness, productivity, or coarse-grained adoption metrics, leaving the role of prompt structure in collaborative integration behavior insufficiently understood. We analyze 265 manually validated developer-ChatGPT interactions derived from self-admitted ChatGPT usage in open-source pull requests. Building on prior research on developer-facing artifacts and prompt engineering, we operationalize prompt structure using three dimensions: Context, Specificity, and Verification. We first evaluate whether LLM-assisted annotation can reliably reproduce human judgments of prompt structure, finding substantial variation across dimensions and workflow contexts. Specificity shows the most stable agreement with human judgments; Context is systematically under-scored by the LLM; and Verification remains difficult to assess consistently, motivating a hybrid human-LLM annotation strategy. Using this validated framework, we then examine how prompt structure influences actionable code generation, code adoption, and integration depth across AI-assisted PR workflows. Specificity and Context are most strongly associated with actionable code generation; Verification emerges as the primary predictor of code adoption; and integration depth is most strongly associated with Context. Overall, our findings show that prompt characteristics exert distinct, stage-dependent effects across AI-assisted software engineering workflows, influencing downstream adoption and integration through contextual grounding, task specificity, and evaluability cues.

Problem

• prompt structure
• pull request outcomes
• LLM-assisted development
• collaborative software engineering
• code adoption

Innovation

• prompt structure
• LLM-assisted development
• pull request outcomes
• hybrid annotation
• stage-dependent effects