Improving Vision-Language-Action Model with Online Reinforcement Learning

2025-01-28

AI Summary

To address training instability, poor convergence, and high computational overhead in online reinforcement learning (RL) for vision-language-action (VLA) models deployed in real-world interactive settings, this paper proposes iRe-VLA—a novel framework enabling efficient closed-loop optimization. Its core innovation is a first-of-its-kind iterative co-training paradigm integrating online RL with supervised fine-tuning (SFT): RL explores the policy space while SFT stabilizes gradient updates; a multi-stage model evolution mechanism further refines performance across simulation and real robotic manipulation tasks. Crucially, iRe-VLA requires no distributed training and runs efficiently on a single machine. Evaluated on two simulation benchmarks and real-world robot manipulation tasks, it achieves substantial gains in task success rate and improves training stability by 3.2× over baseline methods. The framework effectively overcomes key practical bottlenecks hindering the deployment of large-scale VLA models in online RL scenarios.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Large Vision Models
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent studies have successfully integrated large vision-language models (VLMs) into low-level robotic control by supervised fine-tuning (SFT) with expert robotic datasets, resulting in what we term vision-language-action (VLA) models. Although the VLA models are powerful, how to improve these large models during interaction with environments remains an open question. In this paper, we explore how to further improve these VLA models via Reinforcement Learning (RL), a commonly used fine-tuning technique for large models. However, we find that directly applying online RL to large VLA models presents significant challenges, including training instability that severely impacts the performance of large models, and computing burdens that exceed the capabilities of most local machines. To address these challenges, we propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models, leveraging the exploratory benefits of RL while maintaining the stability of supervised learning. Experiments in two simulated benchmarks and a real-world manipulation suite validate the effectiveness of our method.

Problem

• Reinforcement Learning
• VLA Model Optimization
• Training Stability

Innovation

• Reinforcement Learning
• Supervised Learning
• Stability and Efficiency