How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests

2026-06-26

AI Summary

This study addresses a critical gap in software security research by examining how vulnerabilities are communicated in pull requests, moving beyond the common focus on explicitly labeled issues (e.g., CVE identifiers) to include those discussed through implicit security-related language such as “unauthorized access.” Leveraging the AIDev-pop dataset, this work presents the first large-scale empirical comparison of how humans, bots, and AI coding agents employ explicit references and implicit security signals across pull request components—including titles, descriptions, comments, and commit messages. Integrating natural language processing and log analysis, the study correlates these communication patterns with actual vulnerability introduction or remediation, code review dynamics, and merge outcomes. The resulting vulnerability communication behavior map reveals systematic differences among participant types, offering an empirical foundation for enhancing the security awareness of automated development tools.

Abstract

Developers may reference vulnerabilities in pull request discussions through both explicit identifiers, such as CVEs or GHSAs, and implicit security-related language (e.g., "unauthorized access" or "SQL injection"). Prior work has primarily focused on explicit identifiers, potentially overlooking vulnerability discussions that lack formal references. Bots and coding agents are becoming more common in pull requests, raising new questions about how different accounts communicate about vulnerabilities. In this registered report, we describe our planned study of vulnerability communication in pull requests by humans, bots, and coding agents. Building on the AIDev-pop dataset, we analyze explicit vulnerability references and implicit security-related signals across pull request titles, descriptions, review comments, commit messages, and timeline discussions. We further investigate whether these references are associated with vulnerabilities introduced or fixed in the modified code and how they relate to pull request review activity and outcomes. This study contributes a large-scale empirical investigation of vulnerability communication practices in modern software development.

Problem

• vulnerability communication
• pull requests
• security-related language
• bots
• coding agents

Innovation

• vulnerability communication
• pull requests
• security-related language
• bots and coding agents
• empirical software engineering