Turning Video Models into Generalist Robot Policies

2026-05-26

AI Summary

This work addresses the challenge of zero-shot transferring general-purpose video generation models to diverse robotic embodiments for generalized control. The authors propose a decoupled architecture that fixes an action-agnostic video prediction model as a visual planner and separately trains an inverse dynamics model (IDM) grounded in the robot’s Jacobian to map visual trajectories to executable motor commands. The IDM requires only minimal self-play data for efficient training, supports high-dimensional action spaces, and enables zero-shot transfer across distinct robotic morphologies. The approach is validated in both simulation and real-world settings, successfully executing tasks such as object manipulation with a Panda arm and 16-degree-of-freedom cube reorientation with an Allegro hand, thereby significantly enhancing model generalizability and deployment efficiency.

Abstract

Video generative models have emerged as a promising robotics backbone, capable of generating videos that depict the completion of complex tasks across embodiments and environments. Recent work proposes robot foundation models that jointly predict future observations and actions by finetuning video models with action-labeled data. In this paper, we test the limits of an alternative approach: leave the video planner as-is while training an embodiment-specific inverse dynamics model (IDM). This decoupling offers several natural benefits: the video planner remains embodiment-agnostic, different video models can be interchanged easily without re-training the IDM, and the IDM can be independently trained with readily available self-play data. We present a closed-loop, video-to-action policy that combines an action-free video world model with a carefully-designed IDM based on the robot embodiment Jacobian. We demonstrate that our IDM design is both data-efficient and scalable to high-dimensional action spaces. Our policy, which we coin the Video-to-Embodied Robot Action Model (VERA), achieves strong performance across simulated and real-world benchmarks, including zero-shot Panda arm manipulation and 16-DoF Allegro-hand dexterous cube re-orientation. The same video planner can be used across multiple embodiments by pairing it with different embodiment-specific IDMs. Our results show that decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control. More results are available on our project website: https://vera.csail.mit.edu.

Problem

• video models
• robot policies
• embodiment-agnostic
• zero-shot control
• generalizable robot control

Innovation

• inverse dynamics model
• video generative model
• embodiment-agnostic planning
• zero-shot robot control
• Jacobian-based action translation