VITA: Vision-to-Action Flow Matching Policy

2025-07-17

AI Summary

Existing vision-to-action generation methods rely on Gaussian noise priors and conditional mechanisms (e.g., cross-attention), incurring high computational overhead and struggling to model the manifold mapping between visual inputs and sparse, unstructured action sequences. Method: We propose VITA—a novel end-to-end flow matching framework that uses image latent variables as the source distribution. It introduces a purely MLP-based architecture for unconditional vision-to-action mapping, unifying cross-modal manifold learning. A structured action latent space is constructed via an autoencoder, aligned with visual representations through upsampling; action reconstruction is achieved via latent flow decoding and ODE solving under reconstruction supervision. Results: Evaluated on the ALOHA platform, VITA achieves state-of-the-art or superior performance across five simulated and two real-world dual-arm robotic tasks, while reducing inference latency by 50–130%.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

We present VITA, a Vision-To-Action flow matching policy that evolves latent visual representations into latent actions for visuomotor control. Traditional flow matching and diffusion policies sample from standard source distributions (e.g., Gaussian noise) and require additional conditioning mechanisms like cross-attention to condition action generation on visual information, creating time and space overheads. VITA proposes a novel paradigm that treats latent images as the flow source, learning an inherent mapping from vision to action while eliminating separate conditioning modules and preserving generative modeling capabilities. Learning flows between fundamentally different modalities like vision and action is challenging due to sparse action data lacking semantic structures and dimensional mismatches between high-dimensional visual representations and raw actions. We address this by creating a structured action latent space via an autoencoder as the flow matching target, up-sampling raw actions to match visual representation shapes. Crucially, we supervise flow matching with both encoder targets and final action outputs through flow latent decoding, which backpropagates action reconstruction loss through sequential flow matching ODE solving steps for effective end-to-end learning. Implemented as simple MLP layers, VITA is evaluated on challenging bi-manual manipulation tasks on the ALOHA platform, including 5 simulation and 2 real-world tasks. Despite its simplicity, MLP-only VITA outperforms or matches state-of-the-art generative policies while reducing inference latency by 50-130% compared to conventional flow matching policies requiring different conditioning mechanisms or complex architectures. To our knowledge, VITA is the first MLP-only flow matching policy capable of solving complex bi-manual manipulation tasks like those in ALOHA benchmarks.

Problem

• Evolves latent visual representations into actions for visuomotor control
• Eliminates separate conditioning modules in action generation
• Addresses dimensional mismatches between vision and action data

Innovation

• Uses vision-to-action flow matching policy
• Learns mapping from latent images to actions
• Employs autoencoder for structured action space