Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective

2026-05-08

AI Summary

This work addresses the challenge in large language model policy optimization where importance sampling struggles to simultaneously achieve unbiasedness and low variance at both token and sequence levels. The authors propose cumulative token importance sampling ratios, establishing them as the theoretically optimal solution for the first time, and introduce a position-adaptive log-space clipping mechanism. This approach enables unbiased prefix correction and variance reduction within token-level policy gradients. Built upon an off-policy reinforcement learning framework and integrated with tool-augmented reasoning, the method significantly outperforms strong baselines such as GRPO and GSPO across multiple mathematical reasoning benchmarks, achieving state-of-the-art average performance consistently across different model scales.

Abstract

Reinforcement learning, including reinforcement learning with verifiable rewards (RLVR), has emerged as a powerful approach for LLM post-training. Central to these approaches is the design of the importance sampling (IS) ratio used in off-policy policy-gradient estimation. Existing methods face a fundamental bias-variance dilemma: token-level IS ratios, as adopted by PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024), introduce bias by ignoring prefix state distribution mismatch; full sequence ratios provide exact trajectory-level correction but suffer from high variance due to the multiplicative accumulation of per-token ratios, while GSPO (Zheng et al., 2025) improves numerical stability via length normalization at the cost of deviating from the exact full-sequence IS correction. In this work, we identify the cumulative token IS ratio, the product of per-token ratios up to position $t$, as a theoretically principled solution to this dilemma. We prove that, under the token-level policy-gradient formulation, this ratio provides an unbiased prefix correction for each token-level gradient term and has strictly lower variance than the full sequence ratio. Building on this insight, we propose CTPO (Cumulative Token Policy Optimization), which combines the cumulative token IS ratio with position-adaptive clipping that scales log-space clip bounds according to the natural $\sqrt{t}$ growth of the cumulative log-ratio. This yields more consistent regularization across token positions. We implement and evaluate CTPO in the tool-integrated reasoning setting on several challenging mathematical reasoning benchmarks, achieving the best average performance across both model scales compared with strong GRPO and GSPO baselines. Code will be available at https://github.com/horizon-llm/CTPO.

Problem

• importance sampling
• policy optimization
• bias-variance tradeoff
• large language models
• reinforcement learning

Innovation

• importance sampling
• cumulative token ratio
• policy optimization
• LLM reinforcement learning
• variance reduction