MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service

2026-05-08

AI Summary

This work addresses the high cost and poor scalability of reinforcement learning (RL) fine-tuning in multi-user, multi-task settings. To overcome these limitations, the authors propose a multi-tenant asynchronous Reinforcement Learning as a Service (RLaaS) framework that enables parameter-efficient fine-tuning through shared base models augmented with lightweight LoRA adapters. The framework employs a decoupled asynchronous architecture that independently schedules environment interaction, rollout generation, and policy training phases, further reducing inter-task interference via an event-driven mechanism. Experimental results demonstrate that the approach maintains state-of-the-art performance per task even under up to 32 concurrent workloads, achieves a 4.3× improvement in accelerator utilization, and reduces end-to-end training time by 85%.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has significantly improved the reasoning capabilities of large language models (LLMs), particularly in multi-turn agentic settings involving environment interaction like tool use. However, fine-tuning such models remains prohibitively expensive due to high computational requirements, limiting accessibility. We propose MARLaaS (Multi-tenant Asynchronous RL as a Service), a system for concurrent RL fine-tuning across multiple users and tasks. Our approach is based on two key ideas: (1) sharing a base model across tenants using lightweight LoRA adapters, and (2) a disaggregated asynchronous architecture that decouples rollout generation, environment interaction, and policy training into independently scheduled stages. This design enables tasks to progress through the RL pipeline at their own pace in an event-driven manner, reducing cross-task interference, idle time, and end-to-end latency. In multi-task settings (we report up to 32 concurrent tasks), MARLaaS achieves single-task state-of-the-art performance while improving accelerator utilization by up to 4.3x and reducing end-to-end training time by 85%.

Problem

• Reinforcement Learning
• Multi-tenant
• Fine-tuning
• Computational Cost
• Resource Utilization

Innovation

• Multi-tenant RL
• Asynchronous Reinforcement Learning
• LoRA Adapters
• Disaggregated Architecture
• RL as a Service