What Matters for Latent Actions in Robot Learning

2026-08-19

AI Summary

研究通过统一框架评估了41种潜在动作模型设计选择，解决了机器人学习中潜在动作效果不一致的问题，并发现微调视觉-语言模型能有效提升下游策略学习。

Abstract

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ability to reliably predict downstream robotic manipulation performance. Extensive experiments on three widely used benchmarks provide strong empirical evidence that fine-tuning vision-language model (VLM) backbones with latent actions provides a stronger initialization for downstream policy learning, with further validation on real-world robot manipulation tasks.

Problem

• Latent Action Models
• robot learning
• unlabeled videos
• manipulation performance
• design choices

Innovation

• Latent Action Models
• robotic manipulation
• autoencoding framework
• vision-language model backbones