Selected work

Representative Papers

RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian

Jan 03, 2026arXiv.org

This work proposes a fine-tuning-free, large language model (LLM)-driven approach to address the need for high-quality, context-aware, and goal-directed automated code review comments in enterprise settings. By leveraging prompt engineering, contextual retrieval, and a comment quality filtering mechanism, the authors developed and deployed RovoDev Code Reviewer—an integrated system within Atlassian Bitbucket. Evaluation over a one-year period in a real-world industrial environment demonstrates that 38.7% of the system’s automatically generated comments led developers to modify their code, resulting in a 30.8% reduction in average pull request (PR) cycle time and a 35.6% decrease in manual reviewer comments. The system also effectively identified actionable code defects, confirming its practicality and effectiveness without requiring model fine-tuning.

2 citationsRead paper

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Aug 14, 2026

This study addresses the issues of counting scale drift and decision inconsistency arising from the coupling of interpretation and aggregation in multi-source evidence reasoning. We propose a decoupling framework that constructs an evidence quadruplet interface and introduces calibrated log-likelihood ratio pooling to achieve generic rectification at the arithmetic level, thereby effectively separating evidence interpretation from decision aggregation. By integrating sequential encoders with tree ensemble models, the proposed method achieves an AUPRC of 0.921 on longitudinal corpora, significantly outperforming handcrafted baselines (0.805). These results validate the effectiveness of the separation architecture in enhancing the reliability of multi-source reasoning, offering a novel paradigm for complex evidence aggregation.

0 citationsRead paper

EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

Aug 05, 2026

This work addresses a critical gap in existing agent evaluation frameworks, which typically ignore resource costs and thus fail to capture the economic trade-offs inherent in real-world deployment. To bridge this gap, we introduce EcoAgent-Bench—the first benchmark that internalizes budget constraints and action costs as core evaluation dimensions—featuring 304 tasks across five categories derived from GAIA, HotpotQA, and MuSiQue. We evaluate seven LLM-based agents and four control policies in both tool-API and workspace-CLI environments. Introducing an economic consistency score, we expose the limitations of optimizing solely for micro-average accuracy and analyze budget sensitivity via threshold scanning. Our experiments reveal that state-of-the-art agents achieve micro strict success rates of only 3.9–24.0%, with economic consistency peaking at just 7.3%; notably, GPT-5.4 shows negligible improvement in upgrade rate despite increased budgets, highlighting a severe disconnect between task completion capability and economic decision-making.

0 citationsRead paper

Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment

Jul 18, 2026

This work addresses a critical gap in the evaluation of intelligent code review systems, which has predominantly emphasized performance metrics while overlooking agents’ dynamic behaviors, failure modes, and implicit operational costs in real-world developer environments. We propose a trajectory-aware, cost-sensitive evaluation framework that systematically analyzes authentic code review logs from local development settings by integrating trajectory parsing, behavioral modeling, and overhead quantification to assess both planning efficacy and verification costs. Our findings reveal that high-precision reviews often incur substantial exploration and validation overhead, whereas successful cases consistently exhibit stronger upfront planning capabilities that significantly reduce downstream verification burden. This study is the first to incorporate trajectory-level behavior and associated costs into the evaluation paradigm, uncovering key trade-offs essential for the practical deployment of intelligent code review agents.

0 citationsRead paper

Sequential Probability Ratio Test using Z-Statistics (SPRT-z): A Practical Approach for Online Experimentation

Jun 23, 2026

This study addresses critical limitations in current online A/B testing methodologies, where fixed-horizon tests suffer from inflated Type I error due to repeated monitoring, and prevailing sequential approaches struggle to simultaneously support futility stopping, control Type II error, and respect minimum detectable effect constraints. To overcome these challenges, the authors propose SPRT-z, a novel procedure grounded in Wald’s sequential probability ratio test. SPRT-z leverages large-sample normal approximations for computational efficiency, incorporates a scale-free horizon calibration (SFHC) mechanism to preserve statistical power under discrete monitoring, and employs a median-unbiased estimator derived from Brownian motion to correct bias induced by early stopping. Empirical evaluations demonstrate that the method rigorously controls both Type I and Type II errors, substantially reduces required sample sizes, and yields confidence intervals with coverage probabilities closely matching nominal levels.

0 citationsRead paper

Recent publications

Latest Papers

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

Aug 14, 2026

This study addresses the issues of counting scale drift and decision inconsistency arising from the coupling of interpretation and aggregation in multi-source evidence reasoning. We propose a decoupling framework that constructs an evidence quadruplet interface and introduces calibrated log-likelihood ratio pooling to achieve generic rectification at the arithmetic level, thereby effectively separating evidence interpretation from decision aggregation. By integrating sequential encoders with tree ensemble models, the proposed method achieves an AUPRC of 0.921 on longitudinal corpora, significantly outperforming handcrafted baselines (0.805). These results validate the effectiveness of the separation architecture in enhancing the reliability of multi-source reasoning, offering a novel paradigm for complex evidence aggregation.

0 citationsRead paper

EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents

Aug 05, 2026

This work addresses a critical gap in existing agent evaluation frameworks, which typically ignore resource costs and thus fail to capture the economic trade-offs inherent in real-world deployment. To bridge this gap, we introduce EcoAgent-Bench—the first benchmark that internalizes budget constraints and action costs as core evaluation dimensions—featuring 304 tasks across five categories derived from GAIA, HotpotQA, and MuSiQue. We evaluate seven LLM-based agents and four control policies in both tool-API and workspace-CLI environments. Introducing an economic consistency score, we expose the limitations of optimizing solely for micro-average accuracy and analyze budget sensitivity via threshold scanning. Our experiments reveal that state-of-the-art agents achieve micro strict success rates of only 3.9–24.0%, with economic consistency peaking at just 7.3%; notably, GPT-5.4 shows negligible improvement in upgrade rate despite increased budgets, highlighting a severe disconnect between task completion capability and economic decision-making.

0 citationsRead paper

Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment

Jul 18, 2026

This work addresses a critical gap in the evaluation of intelligent code review systems, which has predominantly emphasized performance metrics while overlooking agents’ dynamic behaviors, failure modes, and implicit operational costs in real-world developer environments. We propose a trajectory-aware, cost-sensitive evaluation framework that systematically analyzes authentic code review logs from local development settings by integrating trajectory parsing, behavioral modeling, and overhead quantification to assess both planning efficacy and verification costs. Our findings reveal that high-precision reviews often incur substantial exploration and validation overhead, whereas successful cases consistently exhibit stronger upfront planning capabilities that significantly reduce downstream verification burden. This study is the first to incorporate trajectory-level behavior and associated costs into the evaluation paradigm, uncovering key trade-offs essential for the practical deployment of intelligent code review agents.

0 citationsRead paper

Sequential Probability Ratio Test using Z-Statistics (SPRT-z): A Practical Approach for Online Experimentation

Jun 23, 2026

This study addresses critical limitations in current online A/B testing methodologies, where fixed-horizon tests suffer from inflated Type I error due to repeated monitoring, and prevailing sequential approaches struggle to simultaneously support futility stopping, control Type II error, and respect minimum detectable effect constraints. To overcome these challenges, the authors propose SPRT-z, a novel procedure grounded in Wald’s sequential probability ratio test. SPRT-z leverages large-sample normal approximations for computational efficiency, incorporates a scale-free horizon calibration (SFHC) mechanism to preserve statistical power under discrete monitoring, and employs a median-unbiased estimator derived from Brownian motion to correct bias induced by early stopping. Empirical evaluations demonstrate that the method rigorously controls both Type I and Type II errors, substantially reduces required sample sizes, and yields confidence intervals with coverage probabilities closely matching nominal levels.

0 citationsRead paper

Beyond Single-Agent Alignment: Preventing Context-Fragmented Violations in Multi-Agent Systems

Apr 23, 2026

This work addresses the challenge of collective policy violations in multi-agent systems arising from contextual fragmentation—where agents comply locally yet violate policies globally. To this end, the authors propose Distributed Sentinel, a distributed zero-trust architecture that employs lightweight sidecar proxies to propagate cross-domain security states. The system introduces a novel Semantic Taint Token (STT) protocol and a counterfactual graph simulation mechanism, enabling cross-contextual policy verification without exposing raw data. Evaluated on the authors’ custom benchmark, PhantomEcosystem, Distributed Sentinel achieves an F1 score of 0.95 with an end-to-end latency of only 106 ms, substantially outperforming prompt filtering (F1=0.85) and rule-based DLP approaches (F1=0.65). This study presents the first systematic modeling and effective defense against policy violations induced by contextual fragmentation.

0 citationsRead paper