Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees

2026-01-13

AI Summary

This work addresses the challenge of effectively integrating tool use into long-chain-of-thought (Long CoT) reasoning, a task hindered by scarce training data and high costs of human annotation. The authors propose DART, a novel framework that, for the first time, enables unsupervised discovery and reinforcement of tool-calling behaviors. By constructing dynamic rollout trees and employing a tree-structured advantage estimation mechanism, DART precisely rewards beneficial sub-trajectories involving tool invocation. This approach jointly optimizes multi-step reasoning and tool usage without any human-labeled demonstrations. Evaluated on challenging benchmarks such as AIME and GPQA-Diamond, DART significantly outperforms existing methods, demonstrating its capability to seamlessly integrate tool execution with complex, long-horizon reasoning.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.

Problem

• Tool-Integrated Reasoning
• Chain-of-Thought
• Large Language Models
• Reinforcement Learning
• Long CoT

Innovation

• Tool-Integrated Reasoning
• Chain-of-Thought
• Rollout Trees
• Reinforcement Learning
• Long CoT