Process Reward Model with Q-Value Rankings

2024-10-15

arXiv.org

AI Summary

Existing process reward models (PRMs) treat step-wise evaluation as independent classification tasks, neglecting dependencies among reasoning steps—leading to coarse-grained and suboptimal reward distributions. This work formalizes PRM learning as a Markov decision process (MDP) for the first time and introduces a Q-value-based ranking loss to enable fine-grained, theoretically grounded process-level reward learning. Our method integrates Q-value modeling, multi-policy sampling, process-level supervision within a reinforcement learning framework, and backbone-agnostic model adaptation. On multi-step reasoning benchmarks, it consistently outperforms classification-based PRMs. Ablation studies confirm that the proposed comparative loss is the primary driver of performance gains, significantly improving stability and generalization across diverse language models and sampling strategies.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.

Problem

• Optimizes Q-value rankings for process rewards.
• Addresses interdependencies in sequential decision-making tasks.
• Enhances accuracy in complex reasoning through comparative loss.

Innovation

• Process Q-value Model (PQM)
• Markov Decision Process context
• Comparative loss function optimization