Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

2026-07-03

AI Summary

This study addresses the frequent performance degradation of coding agents following scaffold updates—a phenomenon often misattributed by developers to the underlying large language model (LLM) rather than the scaffold itself. By holding the LLM constant and systematically varying only the scaffold, the authors conduct a longitudinal evaluation across 35 consecutive versions of Qwen Code CLI. Combining the SWE-bench Verified benchmark, open-source version tracking, and pull-request-level qualitative analysis, they establish a causal link between high-frequency scaffold iterations and agent performance fluctuations. The findings reveal that mainstream scaffolds are updated more than twice daily on average, with certain revisions significantly reducing task success rates. The study further identifies specific pull requests that directly impair performance, offering the first empirical guidance for scaffold engineering in LLM-based coding agents.

Abstract

Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these scaffoldings evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after scaffolding updates, yet consistently attribute them to the underlying model rather than the scaffolding itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the scaffolding's contribution. Unlike prior work that fixes the scaffolding and varies the model, we fix the model and vary only the scaffolding, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source scaffoldings (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

Problem

• coding agents
• scaffolding evolution
• agent quality
• large language models
• software engineering tasks

Innovation

• scaffolding evolution
• coding agents
• controlled longitudinal study
• LLM-based software engineering
• agent quality regression