Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index

2026-06-30

AI Summary

This work addresses a fundamental tension in verifiable reward-based reinforcement learning (RLVR) between prioritizing high-entropy tokens and suppressing low-probability ones. To resolve this, the authors propose the Relative Surprise Index (RSI), which jointly models token entropy and selection probability for the first time, yielding an information-theoretic, adaptive selection mechanism termed RSI-S. This approach unifies the paradigms of high-entropy prioritization and low-probability suppression, enabling more robust policy gradient updates. Through theoretical analysis of gradient and entropy dynamics under logit perturbations, RSI-S is seamlessly integrated into existing RLVR algorithms such as GRPO. Experiments on Qwen2.5 models (1.5B–7B parameters) demonstrate consistent improvements, with RSI-S achieving 2–3 percentage points higher avg@32 accuracy than GRPO on the AIME and AMC benchmarks.

Abstract

Reinforcement learning (RL) has become a powerful tool for propelling Large Language Models (LLMs) beyond imitation-based training towards more robust reasoning capabilities. Among existing approaches, RL with Verifiable Rewards (RLVR) has emerged as a pivotal paradigm for advancing LLM reasoning. Despite its empirical success, recent studies have offered different insights. One line of inquiry advocates prioritizing high-entropy token positions during training, while another perspective cautions against allowing low-probability tokens to dominate gradient updates. Notably, although high-entropy tokens are usually correlated with low probability, both paradigms empirically yield substantial performance gains. In this work, we argue that evaluating sampled-token probability or entropy in isolation is insufficient to capture the policy optimization dynamics. To resolve this tension, we introduce the Relative Surprisal Index (RSI), a principled, information-theoretic metric that naturally couples the token's entropy with the probability of the selected token. We show that, under mild conditions, RSI is related to the local ratio between the first-order variations of the logit-gradient norm and predictive entropy under a selected-logit perturbation. Building on RSI, we propose RSI Selection (RSI-S), an entropy-adaptive token filtering method that retains tokens within a stable RSI interval. RSI-S successfully reconciles previous contradictory paradigms and filters out both redundant low-surprisal tokens and unstable high-surprisal tail tokens. Empirical evaluations show that RSI-S achieves higher avg@32 accuracy across different model scales (Qwen2.5-1.5B, 3B, and 7B) on AIME and AMC benchmarks: RSI-S improves avg@32 accuracy by 2--3 percentage points over GRPO. Overall, RSI offers a promising perspective for RLVR improvement.

Problem

• Reinforcement Learning
• Large Language Models
• Token Selection
• RLVR
• Relative Surprisal Index

Innovation

• Relative Surprisal Index
• RLVR
• adaptive token selection
• entropy-aware filtering
• reinforcement learning for LLMs