Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It

2026-06-24

AI Summary

This work addresses the instability in reinforcement learning for multi-step tool use, where abrupt probability surges over control tokens often trigger structural collapse, leading to training instability and sharp performance degradation. The study identifies this failure mechanism and introduces an interleaved training strategy that integrates diverse supervisory signals—including off-policy data, prompt guidance, and error examples—to jointly coordinate supervised fine-tuning and reinforcement learning. Experimental results demonstrate that the proposed approach substantially enhances training stability, recovers tool-use capabilities obscured by strict format constraints, and achieves stronger generalization on in-distribution tasks. However, performance remains limited under out-of-distribution shifts in either format or content. This work establishes a robust training paradigm for complex tool-augmented reasoning.

Abstract

Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.

Problem

• multi-step tool-use
• reinforcement learning collapse
• large language models
• tool-invocation failure
• structured execution disruption

Innovation

• tool-use reinforcement learning
• supervisory signals
• catastrophic collapse
• interleaved training
• out-of-distribution generalization