IRASim: Learning Interactive Real-Robot Action Simulators

2024-06-20

arXiv.org

AI Summary

Existing robotic manipulation methods struggle to model fine-grained robot-object interactions in visual space and lack precise temporal alignment between actions and video frames. To address this, we propose the first interactive video generation paradigm tailored to real-world robotic actions, introducing the dedicated IRASim Benchmark and enabling joint action-visual generation aligned with human preferences. Our approach integrates diffusion models with autoregressive architectures, incorporating multi-stage spatiotemporal modeling, joint training on diverse real-robot datasets, and action-conditioned video synthesis. Experiments across three real-world robotic manipulation datasets demonstrate consistent and substantial improvements over all baselines; human evaluation further confirms significant superiority in perceptual quality and action fidelity. All code, the IRASim Benchmark, and pretrained models are publicly released.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Scalable robot learning in the real world is limited by the cost and safety issues of real robots. In addition, rolling out robot trajectories in the real world can be time-consuming and labor-intensive. In this paper, we propose to learn an interactive real-robot action simulator as an alternative. We introduce a novel method, IRASim, which leverages the power of generative models to generate extremely realistic videos of a robot arm that executes a given action trajectory, starting from an initial given frame. To validate the effectiveness of our method, we create a new benchmark, IRASim Benchmark, based on three real-robot datasets and perform extensive experiments on the benchmark. Results show that IRASim outperforms all the baseline methods and is more preferable in human evaluations. We hope that IRASim can serve as an effective and scalable approach to enhance robot learning in the real world. To promote research for generative real-robot action simulators, we open-source code, benchmark, and checkpoints at https: //gen-irasim.github.io.

Problem

• Model fine-grained robot-object visual interactions accurately
• Enhance action-frame alignment in video generation
• Improve policy evaluation and planning efficiency

Innovation

• Uses diffusion transformer for fine-grained video generation
• Introduces frame-level action-conditioning module
• Enhances action-frame alignment in robot manipulation