Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

2026-02-19

AI Summary

This work addresses the instability in asynchronous reinforcement learning for large language model training, which arises from high policy gradient variance due to stale trajectory data. To mitigate this issue, the authors propose VCPO, a novel method that, for the first time in REINFORCE/GRPO-style algorithms, explicitly links the effective sample size (ESS) to gradient variance. VCPO dynamically scales the learning rate based on ESS and introduces a closed-form, value-network-free minimum-variance baseline, enabling low-overhead, variance-aware control. The approach substantially enhances training stability under asynchronous settings, achieving a 2.5× speedup over baseline methods on mathematical reasoning, general reasoning, and tool-use tasks while matching the performance of synchronous training and outperforming existing stabilization techniques.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reinforcement learning (RL) is widely used to improve large language models on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator markedly $\textbf{higher variance}$: training on stale rollouts creates heavy-tailed importance ratios, causing a small fraction of samples to dominate updates. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and general reasoning benchmarks, we find collapse is reliably predicted by effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a general stabilization method for REINFORCE/GRPO-style algorithms that (i) scales learning rate based on effective sample size to dampen unreliable updates, and (ii) applies a closed-form minimum-variance baseline for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness for asynchronous training across math, general reasoning, and tool-use tasks, outperforming a broad suite of baselines spanning masking/clipping stabilizers and algorithmic variants. This reduces long-context, multi-turn training time by 2.5$\times$ while matching synchronous performance, demonstrating that explicit control of policy-gradient variance is key for reliable asynchronous RL at scale.

Problem

• asynchronous reinforcement learning
• policy gradient variance
• off-policy training
• large language models
• training instability

Innovation

• asynchronous reinforcement learning
• variance control
• policy gradient
• effective sample size
• off-policy optimization