Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Coding

2026-02-27

AI Summary

This study investigates whether less-experienced “ambient coders” in AI-assisted programming environments can effectively substitute for senior developers and examines the impact of such contributions on maintainers’ code review burden. Leveraging a dataset of 22,953 pull requests submitted by 1,719 developers on GitHub, the work presents the first systematic quantification of how developer experience influences pull request quality and collaboration efficiency, based on metrics including contribution scale, reviewer feedback, and acceptance rates. Findings reveal that while low-experience developers produce significantly more code—submitting 2.15 times more pull requests and modifying 1.47 times more files—they impose substantially higher maintenance costs: their submissions receive 4.52 times more review comments, exhibit a 31% lower acceptance rate, and require 5.16 times longer to resolve.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: User Experience and Usability
• Intelligent Robots: Embodied AI

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

AI coding agents allow software developers to generate code quickly, which raises a practical question for project managers and open source maintainers: can vibe coders with less development experience substitute for expert developers? To explore whether developer experience still matters in AI-assisted development, we study $22,953$ Pull Requests (PRs) from $1,719$ vibe coders in the GitHub repositories of the AIDev dataset. We split vibe coders into lower experience vibe coders ($\mathit{Exp}_{Low}$) and higher experience vibe coders ($\mathit{Exp}_{High}$) and compare contribution magnitude and PR acceptance rates across PR categories. We find that $\mathit{Exp}_{Low}$ submits PRs with larger volume ($2.15\times$ more commits and $1.47\times$ more files changed) than $\mathit{Exp}_{High}$. Moreover, $\mathit{Exp}_{Low}$ PRs, when compared to $\mathit{Exp}_{High}$, receive $4.52\times$ more review comments, and have $31\%$ lower acceptance rates, and remain open $5.16\times$ longer before resolution. Our results indicate that low-experienced vibe coders focus on generating more code while shifting verification burden onto reviewers. For practice, project managers may not be able to safely replace experienced developers with low-experience vibe coders without increasing review capacity. Development teams should therefore combine targeted training for novices with adaptive PR review cycles.

Problem

• vibe coding
• novice developers
• code review overhead
• AI-assisted development
• pull request acceptance

Innovation

• AI-assisted development
• vibe coding
• developer experience
• code review overhead
• pull request analysis