QuRL: Efficient Reinforcement Learning with Quantized Rollout

2026-02-15

AI Summary

This work addresses the inefficiency of autoregressive decoding in large language models during reinforcement learning training, which can consume up to 70% of total training time due to slow rollouts. To mitigate this bottleneck, the authors propose a quantized Actor approach that integrates INT8/FP8 quantization, adaptive clipping range (ACR), and invariant scaling within the DeepScaleR and DAPO frameworks. The ACR mechanism effectively prevents training instability over extended periods, while invariant scaling reduces sensitivity to minor weight updates caused by quantization. Experimental results demonstrate that the proposed method accelerates rollouts by 20%–80%, substantially improving overall training efficiency without compromising stability or the efficacy of policy updates.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.

Problem

• reinforcement learning
• large language models
• rollout efficiency
• quantization
• training bottleneck

Innovation

• Quantized Reinforcement Learning
• Adaptive Clipping Range
• Invariant Scaling
• Efficient Rollout
• LLM Training