Ratio-Variance Regularized Policy Optimization

2026-05-26

AI Summary

This work addresses a key limitation of standard policy gradient methods, which rely on heuristic clipping that often truncates high-return yet high-bias updates, thereby degrading gradient signals. To overcome this, the authors propose Ratio Variance Regularization—a technique that constructs a local approximation of the trust region by constraining the variance of policy ratios, replacing hard clipping with a “soft braking” mechanism. This approach preserves informative gradients while effectively leveraging stale data. Formulated within a primal-dual optimization framework, the method unifies on-policy and off-policy updates. Empirical results demonstrate consistent and significant improvements over PPO across seven language models and ten robotic control tasks, with notable gains in sample efficiency and performance—particularly in settings involving mathematical reasoning, sparse rewards, and dynamic environments.

Abstract

Standard on-policy reinforcement learning relies on heuristic clipping to enforce trust regions, but this mechanism imposes a severe cost by indiscriminately truncating high-return yet high-divergence updates. We demonstrate that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints, eliminating the need for binary hard clipping. By acting as a distributional ``soft brake'', this approach preserves critical gradient signals from novel discoveries while naturally down-weighting and enabling the reuse of stale, off-policy data. We introduce ${\bf R}^2{\bf VPO}$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework. Extensive evaluations across $7$ LLM scales, spanning both fast and slow reasoning paradigms, and $10$ robotic control tasks demonstrate the generality of the proposed approach. R$^2$VPO achieves substantial performance gains on mathematical reasoning benchmarks, with particularly pronounced improvements on smaller models, while significantly improving sample efficiency. Furthermore, it consistently outperforms PPO baselines in continuous control domains, particularly in sparse-reward and dynamic environments. Together, these findings establish ratio-variance regularization as a principled foundation for stable and data-efficient policy optimization.

Problem

• trust region
• policy optimization
• ratio variance
• on-policy reinforcement learning
• clipping

Innovation

• ratio-variance regularization
• trust-region constraint
• soft brake
• primal-dual optimization
• sample efficiency