ToolRM: Outcome Reward Models for Tool-Calling Large Language Models

2025-09-15

AI Summary

Existing reward models struggle to reliably evaluate LLMs’ tool-calling capabilities due to overreliance on natural language outputs rather than executable outcomes. Method: We propose a task-outcome-based reward modeling framework: (1) introducing FC-RewardBench—the first benchmark explicitly designed for tool-call evaluation; (2) synthesizing high-quality, commercially licensable training data using open-source LLMs; (3) designing a reward-guided data filtering mechanism; and (4) performing supervised fine-tuning across model scales (1.7B–14B). Crucially, our approach replaces linguistic output supervision with actual tool execution results as the primary reward signal. Contribution/Results: Our method significantly improves cross-domain generalization, outperforming general-purpose baselines by 25% on average across seven diverse benchmarks. It enables efficient data curation and lightweight model adaptation, establishing a novel, trustworthy paradigm for evaluating tool-augmented LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As large language models (LLMs) increasingly interact with external tools, reward modeling for tool use has become a critical yet underexplored area. Existing reward models, trained primarily on natural language outputs, struggle to evaluate tool-based reasoning and execution. To quantify this gap, we introduce FC-RewardBench, the first benchmark designed to systematically assess reward models' performance in tool-calling scenarios. Our analysis shows that current reward models often miss key signals of effective tool use, highlighting the need for domain-specific modeling. To address this, we propose a training framework for outcome-based reward models using data synthesized from permissively licensed, open-weight LLMs. We train models ranging from 1.7B to 14B parameters and evaluate them across seven out-of-domain benchmarks. These models consistently outperform general-purpose baselines, achieving up to 25% average improvement in downstream task performance and enabling data-efficient fine-tuning through reward-guided filtering.

Problem

• Evaluating reward models for tool-based reasoning in LLMs
• Addressing performance gap in tool-calling reward benchmarks
• Developing outcome-based reward models for improved tool execution

Innovation

• Outcome-based reward models for tools
• Training framework with open-weight LLMs
• Evaluated across seven out-of-domain benchmarks