Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning

2026-05-18

AI Summary

Existing tool-integration approaches tightly couple tool invocation with execution, undermining the reasoning coherence and expressive capacity of large language models. This work formally frames the decoupling of tool calling and execution as a novel problem and introduces Implicit Hierarchical GRPO (IH-GRPO), an algorithm that leverages a delayed-execution mechanism and a theoretically derived surrogate loss function to achieve effective hierarchical decision-making without explicitly constructing a high-level controller. Evaluated across six out-of-domain mathematical reasoning benchmarks, IH-GRPO substantially outperforms the strongest baseline, yielding performance gains of 1.87%, 2.16%, and 2.53% on the Qwen3 model series, respectively, while consistently improving results in other domains as well.

Abstract

Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87\%, 2.16\%, and 2.53\% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.

Problem

• tool invocation
• execution decoupling
• mathematical reasoning
• reasoning coherence
• tool-integrated reasoning

Innovation

• decoupled tool invocation
• delayed execution
• hierarchical policy
• surrogate loss
• tool-integrated reasoning