Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

2026-07-11

AI Summary

This work addresses a fundamental geometric inconsistency in reinforcement learning with large language models: the widely used PPO-Clip algorithm employs Euclidean updates that disregard the intrinsic Riemannian geometry of the policy manifold, often leading to premature exploration collapse. The paper is the first to uncover this issue’s geometric origin and proposes Riemannian Isometric Policy Optimization (RIPO), which performs policy updates via isometric transformations aligned with the manifold’s natural geometry. This approach inherently balances exploration and exploitation while preserving policy diversity. Theoretical analysis grounded in bias-variance trade-offs supports the method’s efficacy, and extensive experiments across seven competitive benchmarks demonstrate substantial improvements over existing algorithms—achieving up to a 60% performance gain over GRPO on the AIME24 benchmark.

Abstract

Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).

Problem

• exploration collapse
• PPO-Clip
• Riemannian manifold
• policy optimization
• geometric mismatch

Innovation

• Riemannian manifold
• policy optimization
• exploration collapse
• isometric update
• LLM reinforcement learning