OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review

2026-08-10

AI Summary

Current large language model (LLM)-driven code review systems struggle to balance accuracy and efficiency due to inherent non-determinism, fragmented context, and hallucination. This work proposes an engineering paradigm that injects determinism into otherwise uncertain agents through three stages: scheduling, review, and reflection. Specifically, it employs rule-guided selection of files and standards, grounded ReAct-based tool invocation, parallel sub-agents to handle cross-file dependencies, and a novel asymmetric information-bound independent reflection mechanism to effectively suppress hallucinations. Evaluated on the AACR-Bench, the system significantly outperforms mainstream approaches such as Claude Code and Codex, achieving a peak SEM-F1 score of 25.10%—a 2.17× improvement—across six LLM backends while reducing token consumption by 5–15×.

Abstract

LLM-based code review agents promise scalable, always-on review, yet current systems suffer from two intertwined weaknesses: (1) non-determinism--unbounded tool use makes review outcomes unstable, and (2) context locality--the reviewer's access remains bounded to the diff, capping discoverable issue depth. Both give rise to three challenges: misaligned context retrieval, a coherence-efficiency trade-off in multi-file pull requests, and hallucinated comments that erode trust. To address these, we introduce OpenCodeReview, built on deterministic engineering for uncertain agents: rather than granting maximal freedom, we inject determinism at three deliberate pipeline points. Rule-Guided Dispatch uses a multi-layer rule system to deterministically select files and review criteria, eliminating variability of agent-driven triage. Grounded File Review replaces free-form exploration with a curated tool set exposed through a ReAct loop, while file-level parallel SubAgents balance context coherence against efficiency and recover cross-file dependencies on demand. Independent Reflection introduces a falsification-first filter under an asymmetric information boundary--the reflector sees only the diff, not the agent's tool-augmented exploration--removing hallucinated comments without self-reinforcing bias, improving precision while preserving recall. On AACR-Bench (200 real-world PRs, 10 languages, 1,505 expert-verified comments), OpenCodeReview outperforms mainstream coding agents (e.g., Claude Code and Codex) across six LLM backends, achieving up to 2.17x higher SEM-F1 (25.10% vs. 11.57%) while consuming 5-15x fewer tokens. We open-source OpenCodeReview at https://github.com/alibaba/open-code-review.

Problem

• non-determinism
• context locality
• code review
• hallucination
• LLM agents

Innovation

• deterministic code review
• rule-guided dispatch
• grounded file review
• independent reflection
• agent-based software engineering