TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

2026-08-10

AI Summary

This work addresses the low training throughput, GPU underutilization, and redundant prefill overhead commonly encountered in multi-turn agent reinforcement learning due to environment interactions. The paper proposes the first task-readiness-aware elastic reinforcement learning system, which dynamically coordinates rollout and training resources through Continuous Task Batching (CTB), a Resource-Aware Ref-Actor Pipeline (RA²P), and Elastic Resource Scheduling (ERS). This approach significantly reduces redundant computation and waiting time while preserving algorithmic correctness. Compared to synchronous and asynchronous baselines, it achieves up to 5.6× and over 33% higher training goodput, respectively, improves KV cache hit rate by 1.58×, reduces per-step training time by up to 44.3%, and cuts total waiting time by as much as 77.6%.

Abstract

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, $\textrm{RA}^2\textrm{P}$ selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6$\times$ over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58$\times$, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Problem

• Reinforcement Learning
• Agentic Workloads
• Training Goodput
• Rollout Scheduling
• GPU Utilization

Innovation

• Readiness-Aware Scheduling
• Continuous Task Batching
• Resource-Aware Ref-Actor Pipelining
• Elastic Resource Scaling
• RL Goodput