ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

2026-07-01

AI Summary

Existing software engineering benchmarks evaluate defect repair in isolation, overlooking the contextual dependencies among multiple defects and the challenges of continuous maintenance in real-world scenarios. This work proposes ChainSWE, the first benchmark specifically designed for sequential, dependency-aware defect repair. Built upon six SWE-bench datasets, ChainSWE comprises 54 Python projects and 304 temporally ordered defect chains, enabling the first systematic evaluation of coding agents on multi-turn, context-dependent maintenance tasks. Experimental results reveal that the performance of leading agents degrades by up to 70% as defect chain length increases, exposing significant limitations in their capability for sustained software maintenance and addressing a critical gap in current evaluation frameworks.

Abstract

Language model (LM) agents are increasingly deployed to maintain codebases over extended periods, fixing streams of related defects while carrying context from one fix to the next. Yet existing software engineering (SWE) benchmarks evaluate models one bug at a time: the repository is reset, the codebase is re-read, and a single self-contained issue is graded in isolation. This setting collapses a continuous maintenance workflow into a series of independent sessions, ignoring the cumulative dependencies that make real-world bug fixing challenging. To bridge this gap, we introduce ChainSWE, the first benchmark for evaluating agents on sequential, dependent bug fixes within a shared codebase. We collect chronological chains of 304 issues across 54 Python projects, mined from six SWE-bench-family datasets. Our evaluation across a range of agents and models reveals a consistent performance drop by up to 70% as the chain length increases.

Problem

• software maintenance
• bug fixing
• sequential dependencies
• codebase evolution
• LM agents

Innovation

• ChainSWE
• sequential bug fixing
• code maintenance
• LM agents
• software engineering benchmark