The Lessons of Developing Process Reward Models in Mathematical Reasoning

2025-01-13

AI Summary

This work addresses critical bottlenecks in constructing Process Reward Models (PRMs) for mathematical reasoning—namely, low-quality human annotations and substantial evaluation bias. We first identify a fundamental objective misalignment between Monte Carlo (MC) estimation-based data synthesis and Best-of-N (BoN) evaluation. To resolve this, we propose a consensus filtering mechanism that jointly leverages MC estimates and LLM-as-a-judge to generate high-fidelity process-level supervision signals. Furthermore, we design a multi-granularity evaluation framework integrating both response-level and step-level assessment, enabling fine-grained error localization. Experiments demonstrate that our approach significantly improves PRM accuracy and data efficiency in step-level error detection. The open-sourced PRM achieves state-of-the-art performance across multiple mathematical reasoning benchmarks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Process Reward Models (PRMs) emerge as a promising approach for process supervision in mathematical reasoning of Large Language Models (LLMs), which aim to identify and mitigate intermediate errors in the reasoning processes. However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies. In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods. MC estimation relies on completion models to evaluate current-step correctness, leading to inaccurate step verification. Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs: (1) The unreliable policy models generate responses with correct answers but flawed processes, leading to a misalignment between the evaluation criteria of BoN and the PRM objectives of process verification. (2) The tolerance of PRMs of such responses leads to inflated BoN scores. (3) Existing PRMs have a significant proportion of minimum scores concentrated on the final answer steps, revealing the shift from process to outcome-based assessment in BoN Optimized PRMs. To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task. Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research in building process supervision models.

Problem

• Process Reward Models
• Mathematical Reasoning
• Large Language Models

Innovation

• Consensus Filtering
• Monte Carlo Methods
• Process Reward Models