Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning

2025-10-12

AI Summary

Large language models (LLMs) in reinforcement learning with verifiable rewards (RLVR) commonly suffer from premature policy entropy collapse—degenerating into deterministic policies that impair exploration and reasoning. Method: We propose Adaptive Entropy Regularization (AER), a framework that dynamically balances exploration and exploitation via three mechanisms: (1) difficulty-aware entropy coefficient assignment, (2) initial anchoring to a target entropy, and (3) global policy entropy–driven adaptive coefficient adjustment. Contribution/Results: AER overcomes the sensitivity and instability of fixed entropy coefficients across diverse tasks and LLMs. Evaluated on multiple mathematical reasoning benchmarks, AER significantly improves reasoning accuracy, effectively mitigates entropy collapse, and enhances policy diversity and generalization. It provides a scalable, robust solution for exploration in LLM-based RL, advancing reliability and adaptability in verifiable reward settings.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reasoning ability has become a defining capability of Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a key paradigm to enhance it. However, RLVR training often suffers from policy entropy collapse, where the policy becomes overly deterministic, hindering exploration and limiting reasoning performance. While entropy regularization is a common remedy, its effectiveness is highly sensitive to the fixed coefficient, making it unstable across tasks and models. In this work, we revisit entropy regularization in RLVR and argue that its potential has been largely underestimated. Our analysis shows that (i) tasks of varying difficulty demand distinct exploration intensities, and (ii) balanced exploration may require the policy entropy to be maintained within a moderate range below its initial level. Therefore, we propose Adaptive Entropy Regularization (AER)--a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.

Problem

• RLVR training suffers from policy entropy collapse
• Fixed entropy regularization coefficient causes instability
• Difficulty-aware adaptive entropy balancing is needed

Innovation

• Adaptive coefficient balances exploration and exploitation
• Difficulty-aware allocation adjusts regularization strength dynamically
• Initial-anchored entropy maintains moderate policy entropy range