Group Adaptive Clipping Policy Optimization

2026-08-31

AI Summary

针对固定重要性采样比剪切边界在强化学习中的局限，提出了一种基于群体适应性剪切策略优化方法GAPO，通过自适应调整剪切边界来提高学习效率。

Abstract

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.

Problem

• Reinforcement Learning
• Importance Sampling
• Clipping Boundary
• Policy Optimization
• Learning Signal

Innovation

• Group Adaptive Clipping
• Policy Optimization
• Reinforcement Learning
• Verifiable Rewards
• Reverse-KL Trust-Region