🤖 AI Summary
This work addresses the challenge of reward coupling and credit misassignment in multi-objective reinforcement learning, where conventional methods assign a single advantage value to the entire output sequence. To mitigate this, the authors propose a block-wise advantage estimation method that computes objective-specific advantages independently and applies them only to their corresponding text blocks, effectively decoupling multi-objective optimization. By introducing a group-statistics-based outcome-conditioned baseline, the approach efficiently estimates prefix-dependent advantages for subsequent blocks without requiring additional rollouts. Notably, this is the first block-wise advantage mechanism compatible with Group Relative Policy Optimization (GRPO), enabling automated optimization under verifiable rewards without handcrafted scalar reward design. Evaluated on mathematical reasoning tasks, the method substantially alleviates objective interference, matching the performance of state-of-the-art hand-designed reward schemes while preserving the benefits of confidence-weighted ensembling at test time.
📝 Abstract
Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.