When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR

2026-05-19

AI Summary

This work addresses the critical challenge in reinforcement learning with verifiable rewards (RLVR) where reusing expensive rollout samples often induces policy collapse due to undetected harmful gradient updates. The study identifies and theoretically establishes the “disproportionate weight divergence” (DWD) phenomenon, demonstrating that the gradient norm of the language model head (lm_head) serves as a reliable real-time indicator of policy shift. Building on this insight, the authors propose a lightweight Dynamic Gradient Gating (DGG) mechanism that proactively blocks detrimental gradients before catastrophic policy degradation occurs. Empirical evaluations across mathematical reasoning, ALFWorld, WebShop, and retrieval-augmented question answering benchmarks show that DGG achieves up to 2.93× higher sample efficiency and 2.14× faster training convergence compared to standard reuse strategies, matching or exceeding the performance of single-use baselines.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs), but rollout samples are expensive to obtain, making sample efficiency a critical bottleneck. A natural remedy is to reuse each rollout batch for multiple gradient updates, a standard practice in classical RL. Yet in RLVR, this amplifies policy shift, leading to severe performance degradation. Detecting the onset of degradation early enough to stop reuse remains an open and challenging problem. We close this gap by identifying the \textit{Disproportionate Weight Divergence (DWD)} phenomenon: performance degradation is synchronized with a sharp surge in the \texttt{lm\_head} weight change, while intermediate layers remain stable. Empirically, we verify that DWD emerges consistently across diverse LLMs and tasks. Theoretically, we prove that (i) harmful gradients concentrate at the \texttt{lm\_head} while intermediate layers are structurally attenuated, and (ii) the \texttt{lm\_head} gradient norm lower-bounds the policy divergence. These results establish the \texttt{lm\_head} gradient norm as a principled, real-time signal of catastrophic policy shift. Guided by this insight, we propose \textit{Dynamic Gradient Gating (DGG)}, a lightweight intervention that monitors the \texttt{lm\_head} gradient norm in real time and intercepts harmful gradients before they corrupt the optimizer. DGG consistently matches or exceeds the standard single-use baseline, achieving up to $2.93\times$ sample efficiency and $2.14\times$ wall-clock speedup across math, ALFWorld, WebShop, and search-augmented QA tasks.

Problem

• sample efficiency
• policy shift
• rollout reuse
• performance degradation
• gradient update

Innovation

• Dynamic Gradient Gating
• Disproportionate Weight Divergence
• Sample-Efficient RL
• lm_head Gradient Norm
• Reinforcement Learning with Verifiable Rewards