Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

2026-06-18

AI Summary

This work addresses the instability in large language model (LLM) training for reinforcement learning–based reasoning, often caused by uniform policy updates leading to entropy collapse or excessive exploration. The authors propose the Independent Compositional Tokens (ICT) framework, which shifts the optimization focus from scalar uncertainty to token-level distributional characteristics. By leveraging Jensen–Shannon divergence to identify critical branching points and jointly regulating policy concentration through Shannon entropy and second-order Rényi entropy, ICT enables more precise control over exploration. A distribution-aware token selection mechanism updates only the top 10% most distinctive tokens locally. Evaluated on the Qwen2.5 model series across seven reasoning benchmarks, ICT achieves an average pass@4 improvement of 4.58%—reaching up to 14.9%—significantly outperforming baseline methods such as GRPO, 20-Entropy, and STAPO.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains. To resolve this dichotomy, we introduce the Independent Combinatorial Tokens (ICT) framework, which shifts the optimization focus from scalar uncertainty to the distributional properties of token logits. By leveraging the Jensen-Shannon (JS) divergence between token logits distributions, ICT identifies tokens with distinctive distributional patterns as critical branching points for guiding effective exploration in LLM reasoning. Our theoretical analysis, grounded in both Shannon and second-order Rényi entropy, proves that selectively updating on these tokens regulates policy concentration: it reduces the overall distribution uncertainty measured by Shannon entropy, while controlling probability concentration captured by second-order Rényi entropy. This dual effect prevents over-concentrated token generation from weakening exploration and effectively stabilizes the training landscape. Empirical results demonstrate that updating only the top 10% of unique tokens on Qwen2.5 (0.5B/1.5B/7B) models yields an average pass@4 improvement of 4.58%, with a maximum gain of 14.9%, over GRPO, 20-Entropy, and STAPO baselines across seven benchmarks spanning math, commonsense, and Olympiad-level problems.

Problem

• entropy collapse
• entropy explosion
• LLM reasoning
• optimization instability
• token-level distribution

Innovation

• Independent Combinatorial Tokens
• Jensen-Shannon divergence
• Rényi entropy
• LLM reasoning
• distributional deviation