AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context

2026-01-27

AI Summary

Existing benchmarks for automated code review evaluation are limited by their monolingual scope, lack of repository-level context, and reliance on noisy, incomplete pull request comments as ground truth, which hinders accurate assessment of large language models (LLMs). This work proposes AACR-Bench, the first multilingual evaluation benchmark that provides full cross-file contextual information and employs an innovative “AI-assisted + expert-validated” annotation paradigm, achieving a 285% improvement in defect coverage. Through systematic evaluation of prominent LLMs, the study reveals the critical impact of context granularity, retrieval strategies, and model architecture on review performance, correcting prior misjudgments caused by data limitations. The authors release all data and tools to establish a more rigorous standard for future research in this domain.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

High-quality evaluation benchmarks are pivotal for deploying Large Language Models (LLMs) in Automated Code Review (ACR). However, existing benchmarks suffer from two critical limitations: first, the lack of multi-language support in repository-level contexts, which restricts the generalizability of evaluation results; second, the reliance on noisy, incomplete ground truth derived from raw Pull Request (PR) comments, which constrains the scope of issue detection. To address these challenges, we introduce AACR-Bench a comprehensive benchmark that provides full cross-file context across multiple programming languages. Unlike traditional datasets, AACR-Bench employs an"AI-assisted, Expert-verified"annotation pipeline to uncover latent defects often overlooked in original PRs, resulting in a 285% increase in defect coverage. Extensive evaluations of mainstream LLMs on AACR-Bench reveal that previous assessments may have either misjudged or only partially captured model capabilities due to data limitations. Our work establishes a more rigorous standard for ACR evaluation and offers new insights on LLM based ACR, i.e., the granularity/level of context and the choice of retrieval methods significantly impact ACR performance, and this influence varies depending on the LLM, programming language, and the LLM usage paradigm e.g., whether an Agent architecture is employed. The code, data, and other artifacts of our evaluation set are available at https://github.com/alibaba/aacr-bench .

Problem

• Automated Code Review
• Evaluation Benchmark
• Repository-Level Context
• Multi-Language Support
• Ground Truth Quality

Innovation

• Automatic Code Review
• Repository-Level Context
• AI-Assisted Annotation
• Multi-Language Benchmark
• LLM Evaluation