Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation

2026-04-06

AI Summary

This work addresses the insufficient exploration in reinforcement learning with verifiable rewards (RLVR), where rapid policy convergence often undermines effective learning. The authors observe that conventional entropy regularization exhibits limited efficacy and high sensitivity to hyperparameters when applied to large language models. By analyzing the dynamic behavior of policy entropy, they decompose it into “information entropy,” which preserves diverse solution paths, and “spurious entropy,” which disrupts coherent reasoning patterns. Introducing the novel concept of “entropy refinement,” they uncover a bidirectional entropy modulation mechanism inherent in group-relative advantage estimation. Building on this insight, they propose AsymGRPO, a framework that explicitly applies differential entropy control to positive and negative trajectories. Experiments demonstrate that AsymGRPO significantly outperforms strong baselines and synergistically enhances reasoning performance in RLVR tasks when combined with existing entropy regularization techniques.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning capabilities of large language models (LLMs). However, it faces a fundamental limitation termed \textit{restricted exploration}, where the policy rapidly converges to a narrow set of solutions. While entropy regularization is a popular approach used to sustain exploration, it often proves unreliable for LLMs, suffering from high hyperparameter sensitivity and yielding only marginal performance gains. Motivated by these inefficiencies, we propose to rethink the relationship between policy entropy and exploration. By deriving a parametric formulation of group-relative advantage estimation and analyzing entropy dynamics, we conceptually decompose policy entropy into \textit{informative entropy}, which preserves diverse solution paths, and \textit{spurious entropy}, which erodes reasoning patterns. Our analysis reveals that, in contrast to blind maximization, effective exploration requires \textit{entropy refinement}-a mechanism implicitly embedded in group-relative advantage estimation that sustains informative entropy on positive rollouts while suppressing spurious entropy on negative ones. Guided by this insight, we propose \textbf{AsymGRPO}, an exploratory framework that explicitly decouples the modulation of positive and negative rollouts. This allows for independent control over the preservation of informative entropy and the suppression of spurious noise. Extensive experiments demonstrate that AsymGRPO achieves superior performance compared to strong baselines and exhibits the potential to synergize with existing entropy regularization methods.

Problem

• restricted exploration
• entropy regularization
• reinforcement learning with verifiable rewards
• policy entropy
• large language models

Innovation

• entropy refinement
• asymmetric modulation
• informative entropy
• spurious entropy
• group-relative advantage