RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian

2026-01-03

arXiv.org

AI Summary

This work proposes a fine-tuning-free, large language model (LLM)-driven approach to address the need for high-quality, context-aware, and goal-directed automated code review comments in enterprise settings. By leveraging prompt engineering, contextual retrieval, and a comment quality filtering mechanism, the authors developed and deployed RovoDev Code Reviewer—an integrated system within Atlassian Bitbucket. Evaluation over a one-year period in a real-world industrial environment demonstrates that 38.7% of the system’s automatically generated comments led developers to modify their code, resulting in a 30.8% reduction in average pull request (PR) cycle time and a 35.6% decrease in manual reviewer comments. The system also effectively identified actionable code defects, confirming its practicality and effectiveness without requiring model fine-tuning.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs)-powered code review automation has the potential to transform code review workflows. Despite the advances of LLM-powered code review comment generation approaches, several practical challenges remain for designing enterprise-grade code review automation tools. In particular, this paper aims at answering the practical question: how can we design a review-guided, context-aware, quality-checked code review comment generation without fine-tuning? In this paper, we present RovoDev Code Reviewer, an enterprise-grade LLM-based code review automation tool designed and deployed at scale within Atlassian's development ecosystem with seamless integration into Atlassian's Bitbucket. Through the offline, online, user feedback evaluations over a one-year period, we conclude that RovoDev Code Reviewer is effective in generating code review comments that could lead to code resolution for 38.70% (i.e., comments that triggered code changes in the subsequent commits); and offers the promise of accelerating feedback cycles (i.e., decreasing the PR cycle time by 30.8%), alleviating reviewer workload (i.e., reducing the number of human-written comments by 35.6%), and improving overall software quality (i.e., finding errors with actionable suggestions).

Problem

• code review automation
• large language models
• context-aware
• enterprise-grade
• review-guided

Innovation

• LLM-based code review
• context-aware generation
• review-guided automation
• zero-shot code review
• enterprise-scale deployment