π€ AI Summary
This work addresses a critical gap in the evaluation of intelligent code review systems, which has predominantly emphasized performance metrics while overlooking agentsβ dynamic behaviors, failure modes, and implicit operational costs in real-world developer environments. We propose a trajectory-aware, cost-sensitive evaluation framework that systematically analyzes authentic code review logs from local development settings by integrating trajectory parsing, behavioral modeling, and overhead quantification to assess both planning efficacy and verification costs. Our findings reveal that high-precision reviews often incur substantial exploration and validation overhead, whereas successful cases consistently exhibit stronger upfront planning capabilities that significantly reduce downstream verification burden. This study is the first to incorporate trajectory-level behavior and associated costs into the evaluation paradigm, uncovering key trade-offs essential for the practical deployment of intelligent code review agents.
π Abstract
Agentic code review in terminal-based environments enables early feedback during local development before pull request creation. However, existing evaluations remain performance-centric and fail to capture the dynamic behaviors of repository-grounded agentic reviewers. Understanding these behaviors is critical for identifying how agentic reviewers succeed, fail, and incur hidden operational costs in practice. Then, we analyze the reviewers' behavior based on their trajectories. Our results show that agentic reviewers achieve higher review precision but incur substantial exploration and validation overhead, while successful reviews are associated with stronger planning and less downstream validation. These findings highlight the potential benefits of trajectory-aware and cost-sensitive evaluation of future agentic code review systems.