Learning Action Priors for Cross-embodiment Robot Manipulation

2026-06-24

AI Summary

This work addresses the limitation of existing vision-language-action (VLA) models in lacking explicit motion priors, which hinders their ability to jointly model action temporal dynamics and multimodal alignment across diverse robotic embodiments. To overcome this, the authors propose a two-stage training framework: first, an unconditional action trajectory pretraining stage learns a decoupled action module that captures universal temporal motion structures transferable across robot morphologies; second, this motion prior is integrated into VLA joint training. The approach incorporates a history compression mechanism for efficient temporal context generation and employs a lightweight flow-matching-based encoder-decoder architecture with decoder reuse and early latent distillation. Evaluated on 13 cross-embodiment tasks, the method significantly improves convergence speed and success rates—particularly in data-scarce real-world scenarios—and demonstrates that scaling action data effectively enhances downstream generalization.

Abstract

Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings. In this work, we propose to pretrain the action module with motion priors before cross-modal VLA alignment. Specifically, we introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins. In Stage~1, a lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure solely from unconditioned action trajectories, without processing visual or language tokens. In Stage~2, this learned prior is transferred to VLA training through decoder reuse and early-stage latent distillation, aligning visual-language features with the action embedding space while still allowing end-to-end policy refinement. In addition, the trained encoder serves as a compact history compressor, summarizing state-action histories into a single temporal context token for history-aware modeling at negligible cost. Extensive experiments across 13 diverse cross-embodiment tasks on both simulated and real-world platforms validate the effectiveness of our approach. Compared with VLA training without action priors, our model achieves faster convergence, higher success rates, and substantially stronger performance on data-scarce real-world tasks. Moreover, scaling up the action data in Stage~1 yields a more generalizable action prior that directly improves downstream VLA performance.

Problem

• cross-embodiment
• action priors
• vision-language-action
• motion dynamics
• temporal structure

Innovation

• action priors
• cross-embodiment
• flow-matching
• two-stage training
• temporal motion structure