VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data

2025-02-10

AI Summary

Existing process reward models (PRMs) heavily rely on mathematical reasoning data and exhibit poor cross-domain generalization—particularly on non-mathematical tasks such as legal reasoning. To address this limitation, we propose VersaPRM, the first general-purpose PRM framework designed for multi-domain applicability. Our approach introduces three key innovations: (1) a scalable paradigm for synthesizing reasoning data with fine-grained path-level annotations; (2) a unified PRM architecture supporting supervised fine-tuning across diverse domains; and (3) a reasoning-path-weighted majority voting ensemble mechanism. Evaluated on the legal subset of MMLU-Pro, VersaPRM achieves a 7.9% absolute improvement over strong baselines and significantly outperforms Qwen2.5-Math-PRM (+1.3%). All datasets, code, and trained models are publicly released to foster reproducibility and further research.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Process Reward Models (PRMs) have proven effective at enhancing mathematical reasoning for Large Language Models (LLMs) by leveraging increased inference-time computation. However, they are predominantly trained on mathematical data and their generalizability to non-mathematical domains has not been rigorously studied. In response, this work first shows that current PRMs have poor performance in other domains. To address this limitation, we introduce VersaPRM, a multi-domain PRM trained on synthetic reasoning data generated using our novel data generation and annotation method. VersaPRM achieves consistent performance gains across diverse domains. For instance, in the MMLU-Pro category of Law, VersaPRM via weighted majority voting, achieves a 7.9% performance gain over the majority voting baseline -- surpassing Qwen2.5-Math-PRM's gain of 1.3%. We further contribute to the community by open-sourcing all data, code and models for VersaPRM.

Problem

• Enhance multi-domain reasoning
• Improve non-mathematical performance
• Generate synthetic reasoning data

Innovation

• Multi-domain PRM training
• Synthetic reasoning data generation
• Weighted majority voting enhancement