F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
This work addresses the tendency of group-based sampling in reinforcement learning to overlook rare yet correct trajectories, which biases policies toward frequently occurring solutions. Inspired by Focal Loss, the authors propose a difficulty-aware advantage scaling mechanism that dynamically down-weights updates from high-success-probability samples within group-relative policy optimization frameworks such as GRPO. This approach enhances learning from rare correct trajectories without increasing group size or computational overhead, offering the first explicit solution to the problem of rare correct trajectory forgetting in RLVR. Experiments on Qwen2.5-7B demonstrate significant improvements in pass@256 (e.g., from 64.1 to 70.3 with GRPO) while maintaining or even improving pass@1 performance.