Scheduling Mixed RL Rollouts Beyond Prefix Locality

2026-08-11

AI Summary

This work addresses the intense KV cache contention arising from heterogeneous reinforcement learning rollback tasks—such as RLVR, RLHF, and agent rollouts—in shared inference services, where differences in sequence structure, interaction patterns, and KV cache residency durations degrade throughput efficiency and training objective consistency. To resolve this, the authors propose MISA-T, a routing-layer admission control policy that jointly models workload heterogeneity and KV cache residency characteristics for the first time. MISA-T enables adaptive session admission, workload-aware KV allocation, and residency-aware cache accounting, achieving efficient scheduling while preserving the target task mix ratio. Experiments demonstrate significant improvements: rollback throughput increases by 53.3% and 43.6% on Step3.7 and Qwen3.6-35B-A3B models, respectively, with an average 35.6% throughput gain over 50 rounds, 22.8% reduction in iteration time, and no degradation in task performance.

Abstract

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.

Problem

• reinforcement learning
• rollout scheduling
• KV-cache capacity
• heterogeneous workloads
• prefix locality

Innovation

• mixed rollout scheduling
• KV-cache management
• adaptive admission control
• prefix-aware routing
• reinforcement learning serving