When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation

2026-07-12

AI Summary

This study investigates whether restricting code-generating agents to use only the execute_code tool—instead of broader toolsets—can reduce computational costs without compromising performance, across diverse tasks and agent architectures. Through controlled ablation experiments on synthetic computation tasks and SWE-bench Mini, we compare three configurations: a full baseline, bash-only, and execute_code-only, evaluating both Claude Code and OpenAI Codex CLI agents. Our findings reveal, for the first time, that the cost-effectiveness of tool selection is jointly determined by task mechanics and agent design, and that efficiency should be assessed using cache-adjusted actual costs rather than pass rates alone. In three of four (mechanism, agent) combinations, the execute_code-only setting achieves equal or lower costs with no significant drop in pass rate; the sole cost increase stems from failed trajectories, not from the overhead of successful edits.

Abstract

Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.

Problem

• coding agents
• tool ablation
• code execution
• cost analysis
• task regime

Innovation

• code execution ablation
• tool surface restriction
• cost-efficiency analysis
• coding agent design
• regime-dependent performance