Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis

2026-04-13

AI Summary

This work addresses the challenge of token-level credit assignment under sparse outcome rewards in reinforcement learning, particularly within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, where attributing improvements in reasoning capabilities remains difficult. The study introduces, for the first time, a quadrant-based diagnostic tool that jointly considers reward polarity and token entropy. Leveraging conditional mutual information theory, it formally establishes that a token’s credit is fundamentally bounded by its entropy, thereby exposing how existing methods dilute signals from high-entropy tokens. Building on this insight, the authors propose Entropy-Aware Policy Optimization (EAPO), an algorithm that significantly outperforms strong baselines across two major language models, empirically validating the pivotal role of high-entropy tokens in driving reasoning performance gains.

Technology Category

• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning ability of Large Language Models (LLMs). However, its sparse outcome-based rewards pose a fundamental credit assignment problem. We analyze this problem through the joint lens of reward polarity and token entropy. Our diagnostic tool, the Four Quadrant Decomposition, isolates token updates by polarity and entropy, and controlled ablations show that reasoning improvements concentrate in the high-entropy quadrants. To justify this observation theoretically, we adapt Conditional Mutual Information to the autoregressive RLVR setting and prove that the credit a token can carry is upper-bounded by its entropy. This view yields testable predictions that reasoning gains arise primarily from high-entropy tokens, with unique roles for positive and negative updates. A gradient analysis of GRPO further reveals how uniform reward broadcast dilutes signal at high-entropy positions while over-crediting deterministic tokens. Grounded in these insights, we propose Entropy-Aware Policy Optimization (EAPO) that modulates token-level learning signals accordingly. Extensive experiments demonstrate that EAPO outperforms strong baselines across two model families.

Problem

• credit assignment
• reinforcement learning
• large language models
• token-level optimization
• sparse rewards

Innovation

• credit assignment
• token entropy
• reward polarity
• Entropy-Aware Policy Optimization
• Conditional Mutual Information