Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution

2026-05-07

AI Summary

This work addresses the limitation of existing test evolution benchmarks, which are confined to the method level and thus unable to automatically identify semantically stale or missing test cases at the project level. To bridge this gap, the authors introduce TEBench, the first benchmark specifically designed for project-level test evolution. It encompasses three scenarios—Test-Breaking, Test-Stale, and Test-Missing—and constructs task instances from Defects4J through a four-stage pipeline, providing developer-written ground-truth tests with fine-grained annotations. An evaluation across three agent frameworks and seven configurations involving six base models reveals that all approaches achieve only modest F1 scores (45.7%–49.4%) on test identification, with Test-Stale proving most challenging (F1 ≈ 36%). Moreover, while generated tests are often executable, they frequently deviate in form from the ground truth, exposing a fundamental limitation: current methods overly rely on execution-failure signals.

Abstract

As production code evolves, the test suite must co-evolve to remain effective. Existing benchmarks for test evolution operate at method-level granularity with pre-paired inputs, bypassing the task of locating affected tests from the full project and excluding the need for new tests entirely. We present TEBench, the first project-level benchmark for test evolution. Given a project repository and a code-changing commit, TEBench requires systems to autonomously identify tests requiring modification, determine where new tests are needed, and produce the corresponding test patch. We construct TEBench through a four-stage pipeline over Defects4J projects, curating 314 task instances from 10 projects with developer-written ground truth. Each instance is annotated with one or more of three evolution types: Test-Breaking (tests that fail), Test-Stale (tests that pass but no longer meaningfully validate updated behavior), and Test-Missing (new tests needed for introduced behavior). We evaluate seven configurations spanning three industrial agent frameworks (Claude Code, Codex CLI, OpenCode) and six base models, alongside a heuristic baseline. All seven configurations converge on an identification F1 of 45.7% to 49.4%, revealing a shared performance ceiling across both frameworks and base models. Test-Stale is the most challenging type, averaging F1 around 36%, since configurations rely on execution failure signals and lack proactive semantic reasoning. On the update task, configurations produce highly executable test modifications whose surface form diverges substantially from ground truth. Trajectory analysis reveals a reactive "execute-fail-fix" loop that succeeds for breaking tests but structurally cannot address stale or missing tests. TEBench is available at https://github.com/iSEngLab/TEBench with a leaderboard at https://tebench-leadership.vercel.app.

Problem

• test evolution
• project-level testing
• test maintenance
• software testing
• code evolution

Innovation

• test evolution
• project-level benchmark
• coding agents
• Test-Stale detection
• autonomous test generation