BitsAI-CR: Automated Code Review via LLM in Practice

2025-01-25

AI Summary

To address the limited automation capability of large language models (LLMs) in industrial-scale code review and the difficulty of objectively evaluating feedback adoption rates, this paper proposes a two-stage LLM-driven framework. First, RuleChecker performs initial issue detection using a taxonomy-based rule library; second, ReviewFilter conducts precise validation via structured feedback modeling. We introduce the novel Outdated Rate metric to objectively quantify developers’ actual adoption rate of review comments. Additionally, a data flywheel mechanism enables closed-loop iterative optimization. Experiments show that the framework achieves 75.0% accuracy in generating review comments and maintains a stable Outdated Rate of 26.7% in Go projects. Deployed at ByteDance, it serves over 12,000 weekly active users. This work establishes an evaluable and evolvable industrial paradigm for automated code review.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Code review remains a critical yet resource-intensive process in software development, particularly challenging in large-scale industrial environments. While Large Language Models (LLMs) show promise for automating code review, existing solutions face significant limitations in precision and practicality. This paper presents BitsAI-CR, an innovative framework that enhances code review through a two-stage approach combining RuleChecker for initial issue detection and ReviewFilter for precision verification. The system is built upon a comprehensive taxonomy of review rules and implements a data flywheel mechanism that enables continuous performance improvement through structured feedback and evaluation metrics. Our approach introduces an Outdated Rate metric that can reflect developers' actual adoption of review comments, enabling automated evaluation and systematic optimization at scale. Empirical evaluation demonstrates BitsAI-CR's effectiveness, achieving 75.0% precision in review comment generation. For the Go language which has predominant usage at ByteDance, we maintain an Outdated Rate of 26.7%. The system has been successfully deployed at ByteDance, serving over 12,000 Weekly Active Users (WAU). Our work provides valuable insights into the practical application of automated code review and offers a blueprint for organizations seeking to implement automated code reviews at scale.

Problem

• Code Review
• Large Language Models
• Resource Intensive

Innovation

• AutomatedCodeReview
• RuleCheckerReviewFilter
• ObsolescenceRate