These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests

2026-05-04

AI Summary

This study investigates differences in code review practices between AI-generated and human-authored GitHub pull requests (PRs), challenging the assumption that conventional review metrics adequately reflect human oversight in AI-assisted development. Leveraging the AIDev dataset, the authors conduct a large-scale empirical analysis to systematically compare review behaviors for both types of PRs within the same repositories and categorize interaction patterns between human developers and AI agents. The findings reveal that the majority of AI-generated PRs receive no human review; when reviewed, they are predominantly assessed by AI agents. Humans tend to directly evaluate human-authored PRs but engage indirectly with AI-generated code by guiding AI reviewers rather than performing direct assessment themselves. These results indicate a fundamental shift in code review structures within AI-integrated software workflows.

Abstract

We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset and compare them to human-authored PRs within the same repositories. We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans. Human-authored PRs are more likely to receive human-only review and to attract direct human feedback. In contrast, reviews of AI-generated PRs more often take the form of automation-mediated interaction, with human involvement frequently expressed through agent steering rather than standalone evaluation. These results indicate systematic differences in how review activity is structured in agentic workflows and raise challenges for interpreting review metrics as indicators of human oversight in large-scale mining studies.

Problem

• AI-generated pull requests
• code review
• human oversight
• agentic workflows
• GitHub

Innovation

• AI-generated pull requests
• code review
• human-AI collaboration
• agent steering
• AIDev dataset