WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

2026-07-19

AI Summary

This work addresses the critical bottleneck in synchronous reinforcement learning for long-horizon trajectory generation, where context lengths can exceed tens of thousands of tokens. It introduces workload-awareness into rollout systems for the first time, proposing a joint optimization of decoding and scheduling strategies. During low server load, the method employs model-free SuffixDecoding to speculatively reuse suffixes from historical trajectories; under high load, it leverages cache locality, trajectory progress, and server load to drive global scheduling, thereby reducing redundant KV cache computations and balancing system load. Without modifying the underlying RL algorithm, this approach improves rollout throughput by 1.4× under low load and up to 1.6× under high load, significantly alleviating the efficiency bottleneck in long-context training.

Abstract

Long-horizon rollout generation has become the dominant systems bottleneck in agentic reinforcement learning (RL). As agents interact with environments over many turns, trajectories rapidly grow to tens of thousands of tokens, making synchronous RL training increasingly constrained by rollout. We propose WAR, a workload-aware rollout system that substantially accelerates synchronous agentic RL by jointly optimizing decoding and scheduling. WAR is built on a key observation: the optimal rollout optimization strategy depends on runtime load: (1) Under low load, WAR enables model-free speculative decoding with SuffixDecoding, which reuses suffix patterns from previously completed trajectories as speculative drafts for future rollouts. Unlike model-based drafters, SuffixDecoding introduces no additional draft model and avoids GPU contention with rollout generation. (2) Under high load, where saturated batched decoding leaves limited room for speculative speedup, WAR shifts the optimization focus to cache-aware scheduling. A global scheduler places requests across rollout replicas based on cache locality, trajectory progress and server load, reducing redundant KV-cache recomputation and mitigating load imbalance. By combining decoding-level suffix reuse with system-level rollout scheduling, WAR delivers robust throughput improvements across workload regimes without changing the underlying RL algorithm. WAR improves long-context agentic rollout throughput by 1.4x under low load and up to 1.6x under high load. These results show that WAR removes a major rollout bottleneck in synchronous agentic RL and provides a practical path toward scalable long-context agent training.

Problem

• rollout generation
• synchronous reinforcement learning
• long-horizon trajectories
• systems bottleneck
• agentic RL

Innovation

• speculative decoding
• cache-aware scheduling
• workload-aware optimization
• synchronous reinforcement learning
• long-horizon rollouts