SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

2026-06-11

AI Summary

This work presents the first systematic evaluation of the security robustness of large language model (LLM)-based code review agents against adversarial pull requests (PRs) that simultaneously manipulate both code changes and PR descriptions. To this end, we introduce SEVRA-BENCH, a novel benchmark constructed by reverse-engineering CVE-fixing commits to generate malicious PRs embedding known vulnerabilities, augmented with 15 social engineering tactics—such as appeals to authority and urgency—to craft realistic adversarial examples. Our evaluation across 1,062 malicious PRs on eight prominent LLMs reveals that closed-source models significantly outperform their open-source counterparts, exposing a critical vulnerability in current open-source review agents against social engineering attacks. This study addresses a key gap in existing code security benchmarks by explicitly modeling adversarial code review scenarios.

Abstract

Large language model (LLM) reviewers are increasingly used in pull-request (PR) workflows, where their approvals help decide which code is merged into a repository. This raises a question that benchmarks for static vulnerability detection or code generation do not address: can an automated reviewer reject a malicious contribution when the attacker controls both the code change and the accompanying PR text? We introduce SEVRA-BENCH (Social Engineering of Vulnerabilities in Review Agents), a benchmark that measures how often an automated reviewer approves such adversarial pull requests. Each malicious PR in SEVRA-BENCH is built from a real project commit that previously fixed a vulnerability listed in the Common Vulnerabilities and Exposures (CVE) database. We automatically invert that fix to restore the original vulnerable code and submit it as a pull request wrapped in one of 15 social-engineering framings, which vary the claims made, the supporting evidence, the urgency conveyed, signals of prior approval, and appeals to authority. SEVRA-BENCH contains 1,062 malicious PRs drawn from Common Vulnerabilities and Exposures (CVE)-linked fixes across the top 10 entries of the 2025 Common Weakness Enumeration (CWE) Top 25. In a realistic setting, we evaluate 8 current LLMs as code review agents on PRs that introduce vulnerabilities previously reported in public disclosures. Our results reveal a sharp gap in security capabilities between closed- and open-source models. We hope SEVRA-BENCH will serve as a valuable resource for advancing open-source models and narrowing this gap.

Problem

• social engineering
• vulnerability detection
• automated code review
• adversarial pull requests
• LLM security

Innovation

• social engineering
• automated code review
• LLM security
• adversarial pull requests
• vulnerability benchmark