To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair

2026-06-25

AI Summary

This study addresses the significant resource waste caused by blind code execution in large language model–based program repair, a practice whose cost-effectiveness has lacked systematic evaluation. Leveraging the SWE-bench benchmark, the authors conduct a two-phase empirical analysis of 3,000 repair attempts and 7,745 execution trajectories generated by agents including Claude Code, Codex, and OpenCode under four execution strategies. Their findings reveal that the benefits of code execution are highly concentrated and unevenly distributed, suggesting execution should be treated as a scarce resource requiring careful cost–benefit trade-offs. Notably, disabling execution reduces the repair success rate of top-tier commercial models by only 1.25 percentage points—a statistically insignificant drop—while substantially cutting token consumption and runtime, thereby demonstrating substantial redundancy in current execution practices.

Abstract

LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.

Problem

• code execution
• program repair
• cost-effectiveness
• large language models
• execution behavior

Innovation

• code execution
• program repair
• large language models
• cost-effectiveness
• empirical study