Understanding Diversity Collapse in RLVR via the Lens of Overtraining

2026-06-13

AI Summary

This work addresses the diversity collapse observed in Reinforcement Learning with Verifiable Rewards (RLVR) for enhancing large language models’ reasoning capabilities—where Pass@$1$ improves while higher-$k$ Pass@$k$ deteriorates, narrowing the model’s reasoning frontier. The study identifies this phenomenon as stemming from overtraining: most policy updates concentrate on already saturated problems rather than expanding the boundary of solvable instances. To mitigate this, the authors propose Bayesian Boundary Gating (BBG), a strategy that dynamically prioritizes problems with zero current success to guide effective exploration. Experimental results demonstrate that BBG substantially improves average Pass@$k$ across a wide range of $k$ values, and targeted intervention during training enables Pass@$256$ to surpass baseline performance, confirming the acquisition of genuinely novel reasoning abilities.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from \emph{diversity collapse}: Pass@$1$ improves while high-$k$ Pass@$k$ degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of \emph{overtraining}: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-$k$ Pass@$k$, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-$k$ Pass@$k$, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@$256$ above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose \emph{Bayesian Boundary Gating} (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@$k$ across a wide range of $k$.

Problem

• diversity collapse
• reinforcement learning with verifiable rewards
• reasoning boundary
• overtraining
• Pass@k

Innovation

• diversity collapse
• overtraining
• Bayesian Boundary Gating
• Pass@k
• reinforcement learning with verifiable rewards