BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

2026-06-26

AI Summary

This work addresses the instability of advantage estimation in critic-free reinforcement learning caused by zero within-group reward variance, which hinders cold-start performance. To mitigate this issue, the authors propose a hybrid baseline that integrates semantic clustering–guided historical reward moments with online policy statistics. A confidence-weighting mechanism based on standard error proxies dynamically balances historical and current statistics, while exponential moving averages track cluster-wise reward moments. The approach further incorporates group normalization and PPO-style clipping for adaptive updates. Notably, this is the first method to combine semantic clustering with uncertainty-aware baselines in a critic-free framework. Evaluated on verifiable reasoning benchmarks, it significantly enhances training stability and performance, demonstrating robustness—particularly in binary-reward cold-start scenarios.

Abstract

Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to critic-based PPO pipelines for aligning large language models. However, GRPO-style advantage estimation depends on prompt-local (within-prompt-group) reward statistics and can be unstable. In particular, when all rollouts in a prompt group receive identical rewards, the within-group reward variance becomes zero, and group normalization yields zero advantages for that group, impeding learning in cold-start regimes with binary verifiers. We introduce BV-Blend, a critic-free framework that stabilizes advantage estimation by combining prompt-local on-policy statistics with semantic-cluster-conditioned historical moments. BV-Blend maintains EMA-tracked reward moments for each cluster, derives a confidence weight from a standard error of the mean (SEM) proxy, and uses this weight to blend historical and prompt-local baseline and variance statistics into a standardized advantage for PPO-style clipped updates. Experiments on verifiable reasoning benchmarks show that BV-Blend improves training stability and performance, and remains robust in regimes where group-normalized methods may stall.

Problem

• critic-free reinforcement learning
• advantage estimation
• training instability
• reward variance
• cold-start regime

Innovation

• critic-free RL
• uncertainty-weighted blending
• historical baselines
• semantic clustering
• advantage estimation