🤖 AI Summary
To address two key challenges in Process Reinforcement Learning (PRL)—high computational overhead from external reward models and the absence of a unified theoretical framework for process-level advantage estimation—this paper proposes a Self-Guided Process Reward Optimization (SG-PRO) framework. Methodologically, it theoretically proves that the policy model itself can intrinsically generate process rewards; introduces Cumulative Process Reward and Masked Stepwise Advantage (MSA) to enable fine-grained, step-level advantage estimation under shared prompts, eliminating the need for auxiliary reward models; and integrates policy entropy regularization with response-length constraints. Experiments demonstrate that SG-PRO achieves 3.4× higher training efficiency and 17.5% higher test accuracy than GRPO, reduces average response length by ~33%, and maintains stable training entropy—effectively mitigating reward hacking.
📝 Abstract
Process Reinforcement Learning~(PRL) has demonstrated considerable potential in enhancing the reasoning capabilities of Large Language Models~(LLMs). However, introducing additional process reward models incurs substantial computational overhead, and there is no unified theoretical framework for process-level advantage estimation. To bridge this gap, we propose extbf{S}elf-Guided extbf{P}rocess extbf{R}eward extbf{O}ptimization~( extbf{SPRO}), a novel framework that enables process-aware RL through two key innovations: (1) we first theoretically demonstrate that process rewards can be derived intrinsically from the policy model itself, and (2) we introduce well-defined cumulative process rewards and extbf{M}asked extbf{S}tep extbf{A}dvantage ( extbf{MSA}), which facilitates rigorous step-wise action advantage estimation within shared-prompt sampling groups. Our experimental results demonstrate that SPRO outperforms vaniila GRPO with 3.4x higher training efficiency and a 17.5% test accuracy improvement. Furthermore, SPRO maintains a stable and elevated policy entropy throughout training while reducing the average response length by approximately $1/3$, evidencing sufficient exploration and prevention of reward hacking. Notably, SPRO incurs no additional computational overhead compared to outcome-supervised RL methods such as GRPO, which benefit industrial implementation.