When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

2026-07-08

AI Summary

This work identifies and formally names a previously overlooked issue in actor-only reinforcement learning: the uniform assignment of positive credit to all tokens, which erroneously reinforces low-probability tail tokens and leads to “positive credit contamination,” thereby degrading model reasoning behavior. To address this, the authors propose Tail-aware Credit Optimization (TACO), a novel method that computes a tail-risk score based on local generation context to dynamically modulate credit updates for high-risk tokens. TACO enables fine-grained, context-aware gradient modulation within the GRPO framework without completely suppressing gradients, preserving beneficial rare patterns while mitigating noise. Experiments across three large language models and eight benchmarks demonstrate consistent and significant improvements over the GRPO baseline, enhanced training stability, and sustained performance gains during extended reinforcement learning cycles.

Abstract

Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: https://github.com/xiuyilou/TACO.

Problem

• Reinforcement Learning
• Large Language Models
• Credit Assignment
• Tail Tokens
• Positive-Credit Contamination

Innovation

• Tail-Aware Credit Calibration
• Positive-Credit Contamination
• Uniform Credit Assignment
• Reinforcement Learning for LLMs
• Tail-Risk Score