EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

2026-06-17

AI Summary

This work addresses the high rollout latency in reinforcement learning caused by autoregressive sampling, which is exacerbated by evolving policies and dynamic batching that render conventional speculative decoding ineffective. To overcome these challenges, the paper proposes a system-aware self-speculative decoding framework that automatically constructs a draft model through quantization of the target model—eliminating the need for additional training. The framework dynamically enables or disables speculation based on real-time system states and adaptively adjusts the draft length according to the acceptance rate, thereby aligning with the dynamic nature of rollouts. Experimental results demonstrate that, without compromising final policy performance, the proposed method reduces rollout latency by up to 19.6% and end-to-end latency by 12.7% compared to an autoregressive baseline.

Abstract

Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) offers a natural way to address this bottleneck, as it is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.

Problem

• Reinforcement Learning
• Rollout Generation
• Speculative Decoding
• Latency Bottleneck
• Evolving Policy

Innovation

• speculative decoding
• reinforcement learning rollouts
• system-aware acceleration
• self-speculative decoding
• quantized drafter