Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

2025-05-29

AI Summary

Large Vision-Language Models (LVLMs) suffer from catastrophic forgetting and degraded generalization when applied to embodied robotic control via supervised fine-tuning (SFT). Method: We propose the first end-to-end embodied reasoning framework for robotics based on reinforcement learning—specifically, a PPO variant—that models task-critical state prediction as a rewardable sequential decision-making process conditioned on scene images and expert demonstration metadata, enabling closed-loop reasoning optimization. We innovatively integrate a DeepSeek-R1–inspired reasoning sampling–reinforcement mechanism. Contribution/Results: Our approach achieves, for the first time on a 7B-parameter model, low-level action reasoning performance surpassing GPT-4o. Experiments demonstrate significant accuracy gains over SFT baselines on spatial and fundamental motion reasoning tasks, along with substantially improved generalization and control alignment.

Technology Category

• Intelligent Robots: Embodied AI
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 learns to predict the next keypoint state required for task completion, conditioned on the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions. Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks. Despite having only 7B parameters, Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action control, such as spatial and primitive movement reasoning.

Problem

• Enhancing embodied reasoning for robot control using reinforcement learning
• Overcoming limitations of supervised fine-tuning in robotics tasks
• Improving low-level action control reasoning in robotics

Innovation

• Reinforcement learning enhances robot reasoning
• Predicts next keypoint state from scene metadata
• Outperforms SFT and GPT-4o in action control