Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities

2026-02-04

AI Summary

Standard reinforcement learning in large language model inference often leads to a decline in policy entropy, resulting in mode collapse and insufficient output diversity. This work addresses this issue by introducing a novel diagnostic perspective based on the dynamics of generation probabilities and proposes an Advantage Reweighting Mechanism (ARM). ARM dynamically modulates exploration intensity by jointly modeling prompt perplexity and answer confidence within the reward signal, thereby enhancing reasoning path diversity without compromising accuracy. Experimental results demonstrate that the method improves Pass@1 by 5.7% and Pass@32 by 13.9% on Qwen2.5 and DeepSeek, respectively, effectively mitigating entropy collapse and promoting diverse yet correct reasoning capabilities.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an indispensable paradigm for enhancing reasoning in Large Language Models (LLMs). However, standard policy optimization methods, such as Group Relative Policy Optimization (GRPO), often converge to low-entropy policies, leading to severe mode collapse and limited output diversity. We analyze this issue from the perspective of sampling probability dynamics, identifying that the standard objective disproportionately reinforces the highest-likelihood paths, thereby suppressing valid alternative reasoning chains. To address this, we propose a novel Advantage Re-weighting Mechanism (ARM) designed to equilibrate the confidence levels across all correct responses. By incorporating Prompt Perplexity and Answer Confidence into the advantage estimation, our method dynamically reshapes the reward signal to attenuate the gradient updates of over-confident reasoning paths, while redistributing probability mass toward under-explored correct solutions. Empirical results demonstrate that our approach significantly enhances generative diversity and response entropy while maintaining competitive accuracy, effectively achieving a superior trade-off between exploration and exploitation in reasoning tasks. Empirical results on Qwen2.5 and DeepSeek models across mathematical and coding benchmarks show that ProGRPO significantly mitigates entropy collapse. Specifically, on Qwen2.5-7B, our method outperforms GRPO by 5.7% in Pass@1 and, notably, by 13.9% in Pass@32, highlighting its superior capability in generating diverse correct reasoning paths.

Problem

• Reinforcement Learning
• LLM Reasoning
• Mode Collapse
• Exploration
• Policy Optimization

Innovation

• Advantage Re-weighting Mechanism
• Prompt Perplexity
• Answer Confidence
• Entropy Collapse
• Reinforcement Learning for LLM Reasoning