Selected work

Representative Papers

Coding Agents Don't Know When to Act

May 08, 2026

This study addresses a critical yet overlooked issue in AI-driven code generation: coding agents frequently introduce redundant changes when confronted with already-fixed bug reports, mistaking them as requiring modification and thereby accruing technical debt. The work presents the first systematic characterization and quantification of this “action bias,” introducing FixedBench—a benchmark designed to evaluate agents’ decision-making in scenarios where no code change is necessary. Through human-validated tasks, a multi-agent evaluation framework, and instruction fine-tuning, the authors find that state-of-the-art large language models still produce inappropriate code modifications in 35%–65% of such cases. While failure-reproduction guidance partially mitigates this bias, it simultaneously triggers new failure modes. The paper proposes a novel paradigm that explicitly models “inaction” as a valid and successful outcome, offering a crucial direction for aligning agent behavior with developer intent.

0 citationsRead paper

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Mar 04, 2026

This work addresses the challenge that code generated by large language models (LLMs) often suffers from high complexity, redundancy, and architectural debt, and struggles to autonomously identify and perform human-level refactoring. To this end, we introduce CodeTaste, a benchmark that systematically evaluates LLMs’ ability to detect and reproduce real-world refactorings in multi-file settings. Our approach combines large-scale mining of open-source changes, data-flow analysis, and static pattern detection, employing a two-stage “propose-and-implement” strategy. Refactoring quality is validated through test suites and behavioral equivalence checks. Experiments show that while current LLMs can effectively refactor under explicit instructions, they still exhibit a significant gap in autonomously understanding human refactoring intent. Performance is notably enhanced by the staged strategy and by prioritizing proposals aligned with human practices.

0 citationsRead paper

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Feb 12, 2026

This study presents the first systematic evaluation of the practical impact of repository-level context files—such as AGENTS.md—on the performance of code-generating agents in real-world tasks. By comparing multiple large language model–driven coding agents on the SWE-bench benchmark and a newly curated, human-annotated problem set, with and without access to such context files, the authors find that these files typically reduce task success rates by over 20% and substantially increase computational overhead. The results challenge the assumption that additional contextual information is inherently beneficial, revealing instead that repository-level context often introduces noise and redundancy that impair agent performance. The study recommends retaining only the minimal necessary information to enhance both the efficiency and accuracy of coding agents.

0 citationsRead paper

Automated Benchmark Generation for Repository-Level Coding Tasks

Mar 10, 2025

Existing code agent benchmarks (e.g., SWE-Bench) suffer from high manual construction costs and narrow repository coverage (only 12), leading to distributional bias and poor generalizability to real-world deployment scenarios. To address this, we propose SetUpAgent, the first fully automated benchmark generation system that achieves end-to-end automation of historical dependency installation, test execution, and structured result parsing—enabled by LLM-driven environment reasoning, containerized historical environment reconstruction, and dynamic test suite extraction. Leveraging this system, we release two new benchmarks: SWEE-Bench (spanning ~100 real-world open-source repositories) and SWA-Bench (focused on application-level repair tasks). Empirical evaluation shows that problems in these benchmarks feature lower-quality descriptions and higher repair complexity; consequently, state-of-the-art code agents exhibit up to a 40% drop in success rate. This significantly mitigates distributional shift and yields a more realistic assessment of agent capabilities.

0 citationsRead paper

BaxBench: Can LLMs Generate Correct and Secure Backends?

Feb 17, 2025

This work addresses the lack of rigorous evaluation for large language models’ (LLMs) capability to generate production-grade backend applications. We introduce BaxBench—the first benchmark tailored to secure, multi-file, multi-function web/cloud backend development—comprising 392 tasks spanning mainstream (e.g., Express, FastAPI, Flask) and niche frameworks. Methodologically, we propose an end-to-end evaluation framework integrating functional correctness testing with automated vulnerability exploitation. Our key contributions are: (1) the first systematic assessment of LLMs’ ability to generate complete, functionally correct, and secure backend modules; (2) a novel joint evaluation paradigm combining functional correctness and runtime security exposure; and (3) empirical findings revealing that state-of-the-art models achieve only ~60% functional correctness, over 50% of ostensibly “correct” implementations contain exploitable vulnerabilities, and performance degrades markedly on niche frameworks. BaxBench serves as a critical diagnostic tool for advancing secure autonomous software development.

0 citationsRead paper

Recent publications

Latest Papers

Coding Agents Don't Know When to Act

May 08, 2026

This study addresses a critical yet overlooked issue in AI-driven code generation: coding agents frequently introduce redundant changes when confronted with already-fixed bug reports, mistaking them as requiring modification and thereby accruing technical debt. The work presents the first systematic characterization and quantification of this “action bias,” introducing FixedBench—a benchmark designed to evaluate agents’ decision-making in scenarios where no code change is necessary. Through human-validated tasks, a multi-agent evaluation framework, and instruction fine-tuning, the authors find that state-of-the-art large language models still produce inappropriate code modifications in 35%–65% of such cases. While failure-reproduction guidance partially mitigates this bias, it simultaneously triggers new failure modes. The paper proposes a novel paradigm that explicitly models “inaction” as a valid and successful outcome, offering a crucial direction for aligning agent behavior with developer intent.

0 citationsRead paper

CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

Mar 04, 2026

This work addresses the challenge that code generated by large language models (LLMs) often suffers from high complexity, redundancy, and architectural debt, and struggles to autonomously identify and perform human-level refactoring. To this end, we introduce CodeTaste, a benchmark that systematically evaluates LLMs’ ability to detect and reproduce real-world refactorings in multi-file settings. Our approach combines large-scale mining of open-source changes, data-flow analysis, and static pattern detection, employing a two-stage “propose-and-implement” strategy. Refactoring quality is validated through test suites and behavioral equivalence checks. Experiments show that while current LLMs can effectively refactor under explicit instructions, they still exhibit a significant gap in autonomously understanding human refactoring intent. Performance is notably enhanced by the staged strategy and by prioritizing proposals aligned with human practices.

0 citationsRead paper

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Feb 12, 2026

This study presents the first systematic evaluation of the practical impact of repository-level context files—such as AGENTS.md—on the performance of code-generating agents in real-world tasks. By comparing multiple large language model–driven coding agents on the SWE-bench benchmark and a newly curated, human-annotated problem set, with and without access to such context files, the authors find that these files typically reduce task success rates by over 20% and substantially increase computational overhead. The results challenge the assumption that additional contextual information is inherently beneficial, revealing instead that repository-level context often introduces noise and redundancy that impair agent performance. The study recommends retaining only the minimal necessary information to enhance both the efficiency and accuracy of coding agents.

0 citationsRead paper

Automated Benchmark Generation for Repository-Level Coding Tasks

Mar 10, 2025

Existing code agent benchmarks (e.g., SWE-Bench) suffer from high manual construction costs and narrow repository coverage (only 12), leading to distributional bias and poor generalizability to real-world deployment scenarios. To address this, we propose SetUpAgent, the first fully automated benchmark generation system that achieves end-to-end automation of historical dependency installation, test execution, and structured result parsing—enabled by LLM-driven environment reasoning, containerized historical environment reconstruction, and dynamic test suite extraction. Leveraging this system, we release two new benchmarks: SWEE-Bench (spanning ~100 real-world open-source repositories) and SWA-Bench (focused on application-level repair tasks). Empirical evaluation shows that problems in these benchmarks feature lower-quality descriptions and higher repair complexity; consequently, state-of-the-art code agents exhibit up to a 40% drop in success rate. This significantly mitigates distributional shift and yields a more realistic assessment of agent capabilities.

0 citationsRead paper

BaxBench: Can LLMs Generate Correct and Secure Backends?

Feb 17, 2025

This work addresses the lack of rigorous evaluation for large language models’ (LLMs) capability to generate production-grade backend applications. We introduce BaxBench—the first benchmark tailored to secure, multi-file, multi-function web/cloud backend development—comprising 392 tasks spanning mainstream (e.g., Express, FastAPI, Flask) and niche frameworks. Methodologically, we propose an end-to-end evaluation framework integrating functional correctness testing with automated vulnerability exploitation. Our key contributions are: (1) the first systematic assessment of LLMs’ ability to generate complete, functionally correct, and secure backend modules; (2) a novel joint evaluation paradigm combining functional correctness and runtime security exposure; and (3) empirical findings revealing that state-of-the-art models achieve only ~60% functional correctness, over 50% of ostensibly “correct” implementations contain exploitable vulnerabilities, and performance degrades markedly on niche frameworks. BaxBench serves as a critical diagnostic tool for advancing secure autonomous software development.

0 citationsRead paper