PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR

2026-05-20

AI Summary

This work addresses significant idle time in reinforcement learning with value-based rollouts (RLVR) training, which arises from long-tailed rollouts, tool-calling stalls, and asymmetric resource demands between rollout and training phases—inefficiencies that cannot be eliminated through intra-job optimization. To tackle this, the authors propose PlexRL, a cluster-level runtime system that enables cross-job reuse of a unified large language model (LLM) service. By exploiting the negative correlation of idle periods across jobs, PlexRL employs time-sliced scheduling to fill otherwise unused intervals, thereby avoiding costly model migrations. The system achieves efficient LLM multiplexing under strict affinity constraints through centralized model placement, state transition control, and function-level scheduling. Experiments demonstrate that PlexRL reduces GPU-hour costs by up to 37.58%, substantially increases effective cluster capacity, and incurs minimal per-task overhead while preserving algorithmic flexibility.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algorithms and data. However, RLVR training is notoriously inefficient: long-tailed rollouts, tool-induced stalls, and asymmetric resource requirements between rollout and training introduce substantial idle time that cannot be eliminated by job-local optimizations such as synchronous pipelining, asynchronous rollout, or colocated execution. We argue that this inefficiency is structural. While idle gaps are unavoidable within individual RLVR jobs, they are largely anti-correlated across jobs and therefore exploitable at the cluster level. Leveraging this observation, we present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs. By centrally managing model placement, state transitions, and function-level scheduling under strict affinity constraints, PlexRL time-slices LLM execution across jobs to fill otherwise idle periods without expensive model migration. Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58% while preserving algorithmic flexibility and introducing minimal per-job overhead.

Problem

• Reinforcement Learning with Verifiable Rewards
• LLM Execution Efficiency
• Cluster Resource Utilization
• Idle Time in RLVR
• Asymmetric Resource Requirements

Innovation

• cluster-level orchestration
• multiplexed LLM execution
• RLVR
• time-slicing
• affinity-aware scheduling