🤖 AI Summary
This study addresses the unclear capability of current large language models (LLMs) in code debugging and patch generation within realistic long-context settings (64k–128k tokens). By constructing controlled experiments using the SWE-bench Verified dataset augmented with a human-curated long-context data pipeline and the mini-SWE-agent framework, we systematically evaluate the automatic repair performance of multiple open- and closed-source models. Our work is the first to disentangle the benefits arising from agent-based task decomposition from the models’ intrinsic long-context reasoning abilities, revealing that existing benchmarks fail to effectively assess the latter. Experimental results show a significant drop in repair success rates under ultra-long contexts—for instance, Qwen3-Coder-30B-A3B achieves only 7% and GPT-5-nano 0%—with most agent successes concentrated within the first 20k tokens, underscoring the practical limitations of current models’ usable context capacity.
📝 Abstract
Rapidly increasing context lengths have led to the assumption that large language models (LLMs) can directly reason over entire codebases. Concurrently, recent advances in LLMs have enabled strong performance on software engineering benchmarks, particularly when paired with agentic workflows. In this work, we systematically evaluate whether current LLMs can reliably perform long-context code debugging and patch generation. Using SWE-bench Verified as a controlled experimental setting, we first evaluate state-of-the-art models within an agentic harness (mini-SWE-agent), where performance improves substantially: GPT-5-nano achieves up to a 31\% resolve rate on 100 samples, and open-source models such as Deepseek-R1-0528 obtain competitive results. However, token-level analysis shows that successful agentic trajectories typically remain under 20k tokens, and that longer accumulated contexts correlate with lower success rates, indicating that agentic success primarily arises from task decomposition into short-context steps rather than effective long-context reasoning. To directly test long-context capability, we construct a data pipeline where we artificially inflate the context length of the input by placing the relevant files into the context (ensuring perfect retrieval recall); we then study single-shot patch generation under genuinely long contexts (64k-128k tokens). Despite this setup, performance degrades sharply: Qwen3-Coder-30B-A3B achieves only a 7\% resolve rate at 64k context, while GPT-5-nano solves none of the tasks. Qualitative analysis reveals systematic failure modes, including hallucinated diffs, incorrect file targets, and malformed patch headers. Overall, our findings highlight a significant gap between nominal context length and usable context capacity in current LLMs, and suggest that existing agentic coding benchmarks do not meaningfully evaluate long-context reasoning.