Process Reinforcement through Implicit Rewards

2025-02-03

AI Summary

To address key bottlenecks in LLM reinforcement learning—including the difficulty of online training for process reward models (PRMs), high human annotation costs, and vulnerability to reward hacking—this paper proposes PRIME, an implicit process reward modeling framework that requires no explicit process-level annotations. PRIME eliminates the need for a separate PRM training phase by dynamically updating the PRM online using only policy rollouts and final outcome labels, leveraging self-supervised rollout signals and ensemble-based advantage estimation. It seamlessly integrates with standard RLHF pipelines and is implemented atop Qwen2.5-Math-7B. Experiments demonstrate that PRIME achieves an average 15.1% improvement across seven mathematical and code reasoning benchmarks. Moreover, its lightweight variant, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct across all benchmarks using only 10% of the training data.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs), particularly in tasks requiring complex multi-step reasoning. While dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards, such as training efficiency and credit assignment, this potential remains largely unrealized. This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive, making them particularly vulnerable to reward hacking. To address these challenges, we propose PRIME (Process Reinforcement through IMplicit rEwards), which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards. PRIME combines well with various advantage functions and forgoes the dedicated reward model training phrase that existing approaches require, substantially reducing the development overhead. We demonstrate PRIME's effectiveness on competitional math and coding. Starting from Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several key reasoning benchmarks over the SFT model. Notably, our resulting model, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks with 10% of its training data.

Problem

• Large Language Models
• Complex Thinking Tasks
• Mathematical and Programming Problem Solving

Innovation

• PRIME method
• Simplified information update
• Enhanced performance with less data