What are Key Factors for Updates in RL for LLM Reasoning?

2026-06-21

AI Summary

This work addresses the limitations of existing reinforcement learning from verifiable rewards (RLVR) methods, which rely heavily on heuristic designs and lack a systematic understanding of their update mechanisms. Through theoretical analysis, the study reveals that the number of gradient steps per rollout governs which tokens dominate policy updates by shaping the distribution of importance sampling ratios and influencing clipping behavior, with update dynamics characterized via expected gradients. Building on this insight, the authors propose Adaptive Clipping for Policy Optimization (ACPO), a novel strategy that dynamically adjusts clipping boundaries based on the empirical variance of importance sampling ratios across token groups, thereby enabling more stable and efficient optimization. Experiments on 3B and 7B language models demonstrate that ACPO significantly outperforms strong baselines such as DAPO and CISPO across diverse tasks including mathematical reasoning, tabular question answering, and logical puzzles.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising framework for enhancing the reasoning ability of large language models. However, much of the existing work is guided by heuristic intuition, leading to divergent algorithmic choices, even contradictory ones that nevertheless report empirical gains. To better understand this phenomenon, we conduct a theoretical analysis of RLVR updates. Our study reveals that differences in off-policy degree, determined by the number of gradient steps per rollout, substantially affect the distribution of importance sampling ratios and their clipping behavior, thereby altering which tokens dominate the update. Building on this insight, we characterize gradient expectation as the central quantity governing update dynamics and analyze the roles of token probability, advantage, and importance sampling ratio. Motivated by these findings, we propose Adaptive Clip Policy Optimization (ACPO), which adjusts clipping boundaries across token groups according to the empirical variance of their importance sampling ratios. Experiments on 3B and 7B models across diverse reasoning benchmarks, spanning mathematical problem solving, tabular QA, and logic puzzles, demonstrate that ACPO outperforms strong baselines such as DAPO and CISPO. These results demonstrate that principled, analysis-driven approaches yield more robust and effective RLVR methods. Code is available in: https://github.com/Control-derek/ACPO

Problem

• Reinforcement Learning from Verifiable Rewards
• LLM reasoning
• importance sampling
• clipping behavior
• update dynamics

Innovation

• RLVR
• importance sampling ratio
• adaptive clipping
• off-policy degree
• gradient expectation