Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation

2026-06-23

AI Summary

This work addresses the tight coupling between rollout and training resources in existing reinforcement learning (RL) systems for diffusion-based large generative models, which stems from their reliance on collocated execution and hinders heterogeneous deployment and independent scaling. To overcome this limitation, we propose DigenRL, a decoupled framework enabling flexible resource allocation and efficient scheduling for RL with diffusion-based vision generative models in heterogeneous GPU environments. Key innovations include a Generative Axis Pipeline (GAP), Time-Step Parallelism (TSP), Trainer-Assisted Generation (TAG), and a single-step constrained asynchronous policy, collectively mitigating execution bubbles introduced by decoupling. Experiments demonstrate that DigenRL achieves 1.56–2.10× higher throughput across diverse hardware platforms and mainstream vision generative models, outperforming existing systems such as veRL-Omni and GenRL.

Abstract

Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms, e.g., DanceGRPO and FlowGRPO, have rapidly expanded the scope of RL from language reasoning to diffusion-based visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs remain underexplored. Existing implementations, e.g., veRL-Omni, still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment, and constrains independent scaling. To this end, we introduce DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling. To maximally reduce the execution bubbles in the disaggregated architecture, we propose: 1) a generation-axis pipeline (GAP) and time-step parallelism (TSP) in the diffusion architecture to enable finer-grained pipelining between rollout and training; 2) an elastic trainer-assisted generation (TAG) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations; and 3) a tightly one-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline. Extensive experiments are conducted on three hardware testbeds with 16-32 GPUs using HunyuanVideo-13B, Wan2.1-14B, FLUX.1-12B, and QwenImage-20B generative models. Experimental results show that DigenRL achieves 1.56-2.10x throughput improvements over state-of-the-art diffusion RL systems, veRL-Omni and GenRL.

Problem

• disaggregated reinforcement learning
• diffusion-based generative LLMs
• resource coupling
• heterogeneous deployment
• independent scaling

Innovation

• Disaggregated RL
• Diffusion-Based Parallelism
• Trainer-Assisted Generation
• Time-Step Parallelism
• Generation-Axis Pipeline