Not only where, But when: Temporal Scheduling for RLVR

2026-05-24

AI Summary

Existing RLVR methods suffer from rigid credit assignment due to their inability to dynamically differentiate heterogeneous policy behaviors within a trajectory, which undermines training stability and sample efficiency. This work introduces, for the first time, a temporal scheduling mechanism into RLVR credit assignment that dynamically characterizes policy behavior disparities based on trajectory percentiles. By initially focusing optimization on critical tokens and progressively shifting toward global refinement, the approach enables progressive learning through temporal scheduling, token-level advantage reweighting, and selective optimization. This design fosters a healthier evolution of policy entropy throughout training. Empirical results demonstrate consistent improvements over standard RLVR across mathematical and general reasoning benchmarks, significantly enhancing both training stability and sample efficiency.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a core technique for post-training of Large Language Models (LLMs). While policy optimization is driven by all sampled tokens under a globally broadcast scalar reward, the heterogeneous policy behaviors exhibited along trajectories are largely overlooked without differentiation. Existing works address this by credit allocation, including token-level advantage reweighting, and selective token optimization, however, the allocation criterion are principally stagnant throughout training, limiting resilient policy evolution. In this work, we argue that \textit{when} learning signals are scheduled can be as important as \textit{where} they are allocated across tokens, and introduce the temporal dimension that scheduling the credit allocation criteria over the course of RLVR optimization. We find that prioritizing targeted tokens emphasized with specific policy behaviors, and gradually attenuating toward general optimization leads to more stable and efficient learning dynamics. Furthermore, we show that simple trajectory percentiles provide a natural perspective for distinguishing policy behaviors, and works effectively with temporal scheduling. Our analysis reveals that standard optimization substantially sacrifices policy entropy when simultaneously accommodating heterogeneous behaviors, whereas temporal scheduling yields healthier policy evolution dynamics. Experiments across mathematical and general reasoning benchmarks demonstrate consistent improvements, suggesting that temporal scheduling constitutes a promising optimization dimension.

Problem

• Reinforcement Learning with Verifiable Rewards
• Credit Allocation
• Temporal Scheduling
• Policy Optimization
• Large Language Models

Innovation

• temporal scheduling
• credit allocation
• RLVR
• policy evolution
• trajectory percentiles