DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

2026-05-20

AI Summary

In verifiable reward-based reinforcement learning, sequence-level rewards are difficult to allocate precisely to critical discriminative tokens, causing frequent generic tokens to dominate policy updates and diluting sparse yet informative signals. To address this, this work proposes DelTA, which formulates the policy gradient as a linear discriminant process over token-level gradient vectors. By constructing contrastive gradient centroids from high- and low-reward samples and introducing a token-level coefficient reweighting mechanism, DelTA amplifies discriminative gradient directions while suppressing shared or weakly relevant ones, thereby enabling more accurate credit assignment. Combined with advantage-weighted averaging and a self-normalized RLVR objective, DelTA significantly outperforms the strongest same-scale baselines on seven mathematical reasoning benchmarks—achieving gains of 3.26 and 2.62 points with Qwen3-8B and Qwen3-14B, respectively—and demonstrates strong generalization across code generation, diverse backbone architectures, and out-of-domain tasks.

Abstract

Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose $\textbf{DelTA}$, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.

Problem

• Reinforcement Learning from Verifiable Rewards
• Token Credit Assignment
• Discriminative Learning
• Reward Shaping
• Large Language Models

Innovation

• Discriminative Token Credit Assignment
• Reinforcement Learning from Verifiable Rewards
• Token-level Credit Assignment
• Gradient Reweighting
• Contrastive Policy Update