Revisiting DAgger in the Era of LLM-Agents

2026-05-12

AI Summary

This work addresses the challenge of state distribution shift in long-horizon language model agents, which often stems from early errors during multi-turn interactions. While supervised fine-tuning is limited by covariate shift and reinforcement learning provides only sparse rewards, this study adapts the DAgger algorithm to large language model agents. It collects trajectories via episode-level interpolation between student and teacher policies and leverages dense supervision signals from the teacher to train the student, thereby ensuring consistency with the deployment distribution while enabling efficient learning through real-environment interaction. Evaluated on SWE-bench Verified, the approach improves performance by 3.9 and 3.6 percentage points for 4B and 8B models, respectively—yielding a 27.3% success rate for the 4B agent (surpassing most 8B systems) and 29.8% for the 8B agent, approaching the performance of 32B-scale models.

Abstract

Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory. Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision but suffers from covariate shift because it is trained on off-policy teacher trajectories; while reinforcement learning with verifiable rewards avoids this off-policy mismatch by learning from on-policy rollouts but with only sparse outcome feedback. We address this dilemma by revisiting Dataset Aggregation (DAgger) for multi-turn LM agents: the algorithm collects trajectories through a turn-level interpolation of student and teacher policies, and the student is then trained on these trajectories using supervised labels provided by the teacher. By directly interacting with environments, we expose the model to realistic states likely to be encountered during deployment, thereby effectively mitigating covariate shift. Besides, since the student is learned by mimicking the teacher's behavior, it receives rich feedback during learning. To demonstrate DAgger enjoys the benefits of both worlds, we tested the algorithm to train a software-engineering agent with 4B- and 8B-scale student models. On SWE-bench Verified, our DAgger-style training improves over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B. The resulting 4B agent reaches 27.3%, outperforming representative published 8B SWE-agent systems, while the 8B agent achieves 29.8%, surpassing SWE-Gym-32B and coming within 5 points of stronger 32B-scale agents. Together with consistent gains on the held-out SWE-Gym split, these results suggest the effectiveness of DAgger for modern long-horizon LM agents.

Problem

• long-horizon LM agents
• covariate shift
• multi-turn interaction
• supervised fine-tuning
• reinforcement learning

Innovation

• DAgger
• LLM-agents
• covariate shift
• supervised fine-tuning
• on-policy learning