Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning

2026-06-24

AI Summary

This work addresses the high memory overhead associated with maintaining a library of expert policies in multitask robotic reinforcement learning. It introduces low-rank adaptation (LoRA) into reinforcement learning for the first time, leveraging proximal policy optimization (PPO) to perform parameter-efficient fine-tuning of a shared base policy. The proposed approach constructs a memory-efficient policy library while preserving task success rates without significant degradation. By reducing the memory footprint per policy by 20–160×, it achieves 90–95% storage savings. Consequently, deploying 10–50 distinct policies becomes feasible without requiring memory swapping, substantially enhancing the scalability and practicality of policy libraries in real-world robotic applications.

Abstract

When fine-tuning Large Language Models (LLMs), there has been success in minimizing both memory usage and computation with Parameter-Efficient Fine-Tuning (PEFT), like Low Rank Adaptation (LoRA). In this article, we have explored whether this approach is transferable to the world of robotics and Reinforcement Learning (RL), allowing learning with reduced memory usage and improved computational performance. Specifically, we focused on a version of multi-task robotics, where a library of specialist policies are created. In such a library memory efficiency is especially important. We used a Proximal Policy Optimization (PPO) algorithm and fine-tuned a baseline model to different tasks using LoRA. Our results demonstrate that, depending on the hyperparameters, LoRA can minimize memory usage by a factor of 20-160 compared to full fine-tuning of all layers. This implies a 90-95% storage saving when deploying a library of many (10-50) specialized policies, which can be the differentiating factor between being able to store the entire library in memory or having to use swap-memory in an applied robotics setting. At the same time, our results indicate that there is no significant difference in the success-rate between full fine-tuning and LoRA fine-tuning for the selected tasks.

Problem

• memory efficiency
• policy libraries
• reinforcement learning
• multi-task robotics
• parameter-efficient fine-tuning

Innovation

• Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning
• Reinforcement Learning
• Policy Library
• Memory Efficiency