Spurious Advantage Hidden in GRPO

2026-09-03

AI Summary

论文针对GRPO中的虚假优势问题，提出SIGNBALANCE方法，通过无组合幅度调整来纠正策略优化方向，提高在限定答案任务和搜索代理上的表现。

Abstract

Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.

Problem

• spurious advantage
• reinforcement learning
• reward statistics

Innovation

• SIGNBALANCE
• zero-mean balance
• spurious advantage
• verifiable rewards
• reinforcement learning