Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

2026-05-12

AI Summary

This study addresses the pervasive entropy collapse problem in Reinforcement Learning with Verifiable Rewards (RLVR), which leads to premature policy determinism and training instability. For the first time, we reveal through a token-level entropy flow perspective that entropy-decreasing tokens consistently dominate entropy-increasing ones, causing severe entropy flow imbalance. To mitigate this, we propose an on-policy adaptive entropy flow balancing mechanism (OPEFO) that dynamically regulates entropy flow via a unified interpretive framework and adaptive rescaling of policy gradients, all while preserving strict on-policy characteristics. Experimental results demonstrate that our approach significantly enhances both training stability and final performance across six mathematical reasoning benchmarks.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Existing remedies, including entropy regularization and ratio-based clipping heuristics, either control entropy in a coarse-grained manner or rely on approximate on-policy training. In this paper, we revisit entropy collapse from a token-level entropy flow perspective. Our analysis reveals that entropy-decreasing tokens consistently outweigh entropy-increasing ones, resulting in a severely imbalanced entropy flow. This perspective provides a unified explanation of entropy collapse in existing RLVR algorithms and highlights the importance of balancing entropy dynamics. Motivated by this analysis, we propose On-Policy Entropy Flow Optimization (OPEFO), an adaptive entropy flow balancing mechanism that rescales entropy-increasing and entropy-decreasing updates according to their contributions to entropy change, while remaining strict on-policy. Experiments on six mathematical reasoning benchmarks demonstrate that OPEFO improves training stability and final performance. We will release the code and models upon publication.

Problem

• entropy collapse
• reinforcement learning with verifiable rewards
• on-policy training
• entropy flow
• training instability

Innovation

• entropy flow
• entropy collapse
• on-policy optimization
• RLVR
• adaptive balancing