GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

2026-08-12

AI Summary

This work addresses training instability, cross-task performance degradation, and response length inflation in large language models during rollout-based reinforcement learning post-training by proposing Geometrically Constrained Policy Optimization (GCPO). The method introduces a principal subspace overlap metric to reveal, for the first time, the geometric relationship between single-step policy updates and the dominant singular subspace of pre-trained weights. GCPO enforces hard geometric constraints on policy updates by confining them strictly to the complementary subspace via bilateral orthogonal projection. Experiments demonstrate that GCPO significantly outperforms baseline methods—including GRPO, DAPO, and GSPO—on Qwen3-8B and GLM4-9B, achieving gains of up to 27.69 points in mathematical reasoning, code generation, and tool-use tasks, while effectively mitigating response inflation and preserving both generalization capability and policy entropy stability.

Abstract

On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.

Problem

• rollout RL
• training instability
• subspace geometry
• capability degradation
• response-length inflation

Innovation

• subspace geometry
• Principal-Subspace Overlap
• orthogonal projection
• rollout RL
• GCPO