Steer, Don't Solve: Training Small Critic Models for Large Code Agents

2026-06-19

AI Summary

This work addresses the challenge that large language model–based code agents struggle to efficiently acquire strategic reasoning capabilities through end-to-end training, a process that is both ineffective and computationally expensive. To overcome this limitation, the authors propose freezing the primary agent and introducing a lightweight critic model that delivers real-time, fine-grained supervisory feedback during trajectory execution, thereby guiding the agent to refine its decision-making rather than directly producing final answers. The critic is trained via supervised fine-tuning and demonstrates strong cross-model transferability—evidenced by successful deployment with both CWM-32B and Qwen-family models. Evaluated on SWE-bench Verified, the approach improves accuracy by 3.0–5.2 percentage points over baseline methods, achieving 25.2% accuracy compared to Qwen3-Next-80B-A3B alone while reducing inference cost to $0.04.

Abstract

End-to-end code agent training is resource-intensive and plateaus on the strategy-level reasoning needed to resolve code issues, since jointly optimizing code-level execution and strategy-level reasoning leaves the latter underdeveloped. Instead, we freeze the agent and add a critic model to supply that signal. Prior code critics are post-hoc, scoring completed trajectories rather than steering the agent; we instead train a small critic that provides intra-trajectory feedback via Supervised Fine-Tuning. On SWE-bench Verified, a critic trained on CWM-32B trajectories transfers to two unseen agents (gains of +3.0 to +3.8 points), and adding target-agent trajectories to the corpus increases the gain to +3.8 on CWM-32B and +4.4 to +5.2 on two Qwen agents, at 30-92x lower critic cost than a strong teacher. On Qwen3-Next-80B-A3B, the critic-guided system is both more accurate (25.2% vs. 20.8%) and cheaper (\$0.04 vs. \$0.11) than the agent alone, because the critic also shortens trajectories. Our results show that a small, well-trained critic is a practical complement to scaling agent training. Code: https://github.com/shubhamrgandhi/critic-training. Data and models: https://huggingface.co/collections/shubhamrgandhi/critic-training-for-code-agents

Problem

• code agents
• strategy-level reasoning
• critic model
• end-to-end training
• code generation

Innovation

• critic model
• code agent
• supervised fine-tuning
• trajectory steering
• strategy-level reasoning