Code Review Agent Benchmark

2026-03-24

AI Summary

This work addresses the lack of effective benchmarks for evaluating the code review capabilities of AI systems amidst their growing prevalence. We propose c-CRAB, the first systematic evaluation benchmark constructed from real-world human code review records. Leveraging automatically generated test cases and an automated assessment framework, c-CRAB enables quantitative evaluation of both open-source (e.g., PR-Agent) and commercial (e.g., Devin, Claude Code, Codex) code review agents. Our experiments reveal that current agents achieve an overall task success rate of only approximately 40%. Furthermore, we observe complementary perspectives between human and AI reviewers, and demonstrate that agent-generated tests can serve as an independent quality gate, offering a promising avenue for future human-AI collaborative code review.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-AI Collaboration / Human-AI Teaming

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Software engineering agents have shown significant promise in writing code. As AI agents permeate code writing, and generate huge volumes of code automatically -- the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases -- the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today -- the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex. Our c-CRAB dataset is systematically constructed from human reviews -- given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. Firstly, the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews -- indicating the potential for human-agent collaboration for code review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents -- remains to be investigated.

Problem

• code review
• AI agents
• benchmark
• code quality
• pull request

Innovation

• code review benchmark
• AI software engineering agent
• automated code quality assurance
• human-AI collaboration
• c-CRAB dataset