ViPRA: Video Prediction for Robot Actions

2025-11-11

AI Summary

This paper addresses robot continuous control from unlabeled video demonstrations. We propose a motion-aware latent action representation coupled with a chunked flow-matching decoder. Our method explicitly models *what changes* and *how it changes* in visual dynamics, jointly optimizing a perception loss and optical flow consistency constraint within a pretrain-fine-tune framework to enable cross-morphology generalization. With only ~100 demonstration videos, it achieves smooth, high-frequency (22 Hz) continuous control. On the SIMPLER benchmark, our approach improves performance by 16%; on real-world manipulation tasks, it yields a 13% gain; and it supports cross-platform transfer. Key contributions are: (1) a motion-centric latent action representation that captures spatiotemporal dynamics directly from visual change; and (2) a lightweight, efficient flow-matching decoder explicitly regularized by optical flow constraints to ensure physically plausible action decoding.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We will release models and code at https://vipra-project.github.io

Problem

• Learning robot control from actionless videos without expensive annotations
• Predicting future visual observations and motion-centric latent actions
• Mapping latent actions to robot-specific continuous control sequences

Innovation

• Predicts future observations and latent actions
• Uses perceptual losses for physically grounded behavior
• Maps latent actions to robot control via flow matching