Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub

2026-01-01

arXiv.org

AI Summary

This study presents the first systematic characterization of the practical role of autonomous coding agents in software security. Leveraging the AIDev dataset, we conduct a large-scale empirical analysis of over 33,000 agent-generated pull requests (PRs) on GitHub, identifying and manually validating 1,293 security-related PRs. Through keyword filtering, manual validation, open coding, and meta-analysis of code reviews, we find that security-related PRs constitute approximately 4% of all agent-generated contributions, exhibit lower merge rates, and undergo slower review processes. Their primary contributions lie in security hardening—such as improvements to testing, documentation, configuration, and error handling—rather than direct vulnerability fixes. Moreover, PR complexity and verbosity emerge as key predictors of rejection, exerting a significantly stronger influence than the security-related nature of the contribution itself.

Technology Category

• Philosophy and Ethics of AI: Privacy & Security
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Autonomous coding agents are increasingly deployed as AI teammates in modern software engineering, independently authoring pull requests (PRs) that modify production code at scale. This study aims to systematically characterize how autonomous coding agents contribute to software security in practice, how these security-related contributions are reviewed and accepted, and which observable signals are associated with PR rejection. We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset, comprising of over 33,000 curated PRs from popular GitHub repositories. Security-relevant PRs are identified using a keyword filtering strategy, followed by manual validation, resulting in 1,293 confirmed security-related agentic-PRs. We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes. Moreover, we apply qualitative open coding to identify recurring security-related actions and underlying intents, and examine review metadata to identify early signals associated with PR rejection. Security-related Agentic-PRs constitute a meaningful share of agent activity (approximately 4\%). Rather than focusing solely on narrow vulnerability fixes, agents most frequently perform supportive security hardening activities, including testing, documentation, configuration, and improved error handling. Compared to non-security PRs, security-related Agentic-PRs exhibit lower merge rates and longer review latency, reflecting heightened human scrutiny, with variation across agents and programming ecosystems. PR rejection is more strongly associated with PR complexity and verbosity than with explicit security topics.

Problem

• AI teammates
• autonomous coding agents
• software security
• pull requests
• GitHub

Innovation

• agentic pull requests
• AI teammates
• software security
• empirical study
• code review