Predicting Acceptance and Review Effort in Human and Agent Pull Requests

2026-07-13

AI Summary

This study addresses the challenge faced by project maintainers in predicting the likelihood of pull request (PR) acceptance and associated review effort when code is submitted collaboratively by human developers and AI coding agents. The authors propose a leakage-free prediction framework that leverages only textual features, metadata, repository context, temporal signals, and lightweight diff statistics available at PR submission time to systematically compare the predictability of PRs authored by humans versus AI agents. Using the AIDev dataset, they evaluate logistic regression, tree-based models, and multilayer perceptrons across multiple modeling perspectives. Results show that tree-based models achieve F1 scores exceeding 0.95 for PR acceptance prediction, whereas review effort—measured by comment count and time-to-merge—is only weakly explained by submission-time features, underscoring the influence of team-specific practices. The work further reveals, for the first time, the critical role of textual clarity and metadata in predicting PR acceptance.

Abstract

Pull requests (PRs) are a central mechanism for reviewing and integrating code changes in modern software repositories. As AI coding agents begin to submit more code changes alongside human developers, maintainers face a new challenge: deciding which PRs are likely to be accepted and which ones may require substantial review effort. This paper studies whether such outcomes can be estimated at the time a PR is opened, before reviewer discussion, CI feedback, or merge decisions are available. Using the AIDev dataset, we construct a leakage-aware prediction pipeline for human- and agent-authored PRs. The feature set is limited to submission-time information, including PR text characteristics, metadata, repository context, temporal signals, and lightweight diff statistics. We evaluate classical machine-learning models, including Logistic Regression, Random Forests, Gradient Boosting, Extra Trees, and MLPs, across pooled, human-only, agent-only, and balanced contributor views. Our results show that acceptance prediction is feasible from early signals: tree-based models achieve F1 scores above 0.95, with textual clarity and metadata among the most influential predictors. Review-effort prediction is more difficult. Comment counts and time-to-merge are only modestly explained by submission-time features, suggesting that reviewer availability, project workflow, and team-specific review practices play a major role. These findings indicate that early PR models can support triage and reviewer prioritization, but should be used as advisory tools rather than automated decision-makers.

Problem

• pull requests
• acceptance prediction
• review effort
• AI coding agents
• software repositories

Innovation

• pull request prediction
• AI coding agents
• early signal modeling
• review effort estimation
• leakage-aware pipeline