ToolRL: Reward is All Tool Learning Needs

2025-04-16

AI Summary

Current large language models (LLMs) rely on supervised fine-tuning (SFT) for tool-use learning, exhibiting poor generalization; reinforcement learning (RL) approaches are hindered by coarse-grained rewards (e.g., final answer matching), failing to guide fine-grained tool selection and parameter invocation. Method: We propose the first multi-dimensional reward design framework tailored for tool-calling tasks, systematically characterizing reward types, granularity, and temporal structure to establish a principled, fine-grained reward mechanism. Leveraging Group Relative Policy Optimization (GRPO), we enable end-to-end RL training for tool calling. Contribution/Results: Our method achieves significant improvements—+15% over SFT baselines and +17% absolute gain across multiple benchmarks—while demonstrating enhanced training robustness, scalability, and stability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.

Problem

• Improving generalization of LLMs in unfamiliar tool use scenarios
• Addressing challenges in reward design for tool selection and application
• Enhancing tool use capabilities of LLMs through principled reward strategies

Innovation

• Comprehensive reward design for tool selection
• Group Relative Policy Optimization training
• Fine-grained feedback for tool use tasks