Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners

2026-06-28

AI Summary

This work addresses three structural challenges—channel contamination, granularity mismatch, and cumulative traps—that arise when integrating dense signals with the GRPO framework in process-supervised reinforcement learning for large language models. To resolve these issues, the authors propose PASS, a middleware that reshapes arbitrary scalar step-level process signals through a three-stage mechanism: advantage fusion, value-based chunking, and length normalization, thereby improving credit assignment. PASS is the first approach to systematically identify and simultaneously mitigate all three pathologies, offering paradigm-agnostic and plug-and-play compatibility with diverse process signals such as PRMs and KL distillation. Empirical results demonstrate consistent improvements in pass@1 performance across mathematical reasoning and multi-hop question answering tasks, under two distinct signal paradigms and group normalization operators.

Abstract

Group Relative Policy Optimization (GRPO) is a default recipe for process-supervised reinforcement learning of LLM reasoners, and dense process supervision -- via learned process reward models (PRMs) or on-policy-distillation KL signals -- is a common way to densify its otherwise weak outcome reward. Layering such a step-level signal on top of GRPO's group-standardized advantage, however, exposes three structural pathologies: \emph{channel contamination} between the pooled process, outcome, and format streams at group standardization; \emph{resolution mismatch} between the granularity of the process signal and the granularity of the logical decisions being credited; and a \emph{cumulative trap} by which GRPO's return-to-go sum surfaces either length inflation or truncated exploration depending on the sign regime of the signal. We propose \textbf{PASS} (\emph{Process Advantage Signal Shaping}), a compact middleware that sits between any scalar step-level process signal and GRPO's clipped surrogate and addresses the three pathologies in turn: \emph{Advantage Fusion} standardizes the three streams independently within each group, \emph{Chunk-by-Value} derives value-homogeneous chunks from the signal itself and broadcasts credit within each chunk, and \emph{Divide-Length} converts the cumulative objective into an average-value-density score. We validate PASS across two domains and two process-signal paradigms -- a learned PRM on mathematical reasoning and an on-policy-distillation KL signal (with a generalized variant) on multi-hop question answering -- and under two group-standardization operators. In every regime PASS delivers a consistent pass@1 gain over the corresponding GRPO baseline.

Problem

• process-supervised RL
• GRPO
• advantage estimation
• signal shaping
• LLM reasoning

Innovation

• Process Advantage Signal Shaping
• Group Relative Policy Optimization
• Process Supervision
• Reinforcement Learning
• LLM Reasoning