Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning

2025-04-21

AI Summary

This paper addresses the reward hacking problem in Process Reward Modeling (PRM) during reinforcement fine-tuning, caused by conventional sum-form credit assignment. We propose PURE, a novel framework centered on a min-form value function—defining state value as the *minimum* future reward—to fundamentally suppress strategic manipulation by large language models toward high-reward steps. We provide the first theoretical analysis revealing that additive value functions are the primary cause of early training collapse and reward cheating in PRM. PURE integrates PRM-guided RLHF with hybrid reward fine-tuning (90% process reward + 10% verifiable reward). On Qwen2.5-Math-7B, PURE achieves 82.5% accuracy on AMC23 and an average of 53.3% across five benchmarks. It matches the performance of purely verifiable-reward methods using only 30% of the reasoning steps, whereas sum-form baselines fail entirely.

Technology Category

• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Process reward models (PRMs) have proven effective for test-time scaling of Large Language Models (LLMs) on challenging reasoning tasks. However, reward hacking issues with PRMs limit their successful application in reinforcement fine-tuning. In this paper, we identify the main cause of PRM-induced reward hacking: the canonical summation-form credit assignment in reinforcement learning (RL), which defines the value as cumulative gamma-decayed future rewards, easily induces LLMs to hack steps with high rewards. To address this, we propose PURE: Process sUpervised Reinforcement lEarning. The key innovation of PURE is a min-form credit assignment that formulates the value function as the minimum of future rewards. This method significantly alleviates reward hacking by limiting the value function range and distributing advantages more reasonably. Through extensive experiments on 3 base models, we show that PRM-based approaches enabling min-form credit assignment achieve comparable reasoning performance to verifiable reward-based methods within only 30% steps. In contrast, the canonical sum-form credit assignment collapses training even at the beginning! Additionally, when we supplement PRM-based fine-tuning with just 10% verifiable rewards, we further alleviate reward hacking and produce the best fine-tuned model based on Qwen2.5-Math-7B in our experiments, achieving 82.5% accuracy on AMC23 and 53.3% average accuracy across 5 benchmarks. Moreover, we summarize the observed reward hacking cases and analyze the causes of training collapse. Code and models are available at https://github.com/CJReinforce/PURE.

Problem

• Addresses reward hacking in process reward models (PRMs).
• Proposes min-form credit assignment to improve reinforcement learning.
• Enhances reasoning performance with limited verifiable rewards.

Innovation

• Min-form credit assignment replaces summation
• Limits value function range effectively
• Distributes advantages more reasonably