On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse

2026-06-28

AI Summary

This work investigates Group Relative Policy Optimization (GRPO), revealing that its use of intra-group average return as a baseline imposes a zero-sum constraint on the advantage function, which undermines credit assignment and induces gradient sparsity, thereby hindering multi-step reasoning. Building upon the policy gradient theorem, the study is the first to demonstrate that the GRPO gradient matrix possesses an intrinsic rank-2 structure, proving that its effective rank remains approximately two regardless of group size. It further establishes that this baseline is optimal under specific conditions and quantifies the exacerbation of gradient sparsity during training. Through theoretical analysis, singular value decomposition, and empirical validation on Nemotron-4B with GSM8K, the paper identifies credit assignment as the critical bottleneck limiting multi-step reasoning performance.

Abstract

Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, revealing a fundamental credit assignment failure: under output-only reward, every token in a rollout receives identical advantage, collapsing token-level credit to a single scalar. We prove this induces gradient sparsity that intensifies over training, and demonstrate empirically via SVD analysis of GRPO gradients on Nemotron-4B/GSM8K that the gradient matrix has effective rank $\approx$ 2 regardless of group size $R \in \{2, 4, 8\}$. We formalize this as an intrinsic rank-2 structure arising from the zero-sum constraint on advantages and derive conditions under which GRPO's baseline is optimal. Our results characterize when GRPO's simplicity is theoretically justified and identify the credit assignment bottleneck as the key limitation for multi-step reasoning.

Problem

• credit assignment
• gradient sparsity
• rank collapse
• policy optimization
• reinforcement learning

Innovation

• credit assignment
• gradient sparsity
• rank collapse
• policy gradient
• Group Relative Policy Optimization