Lift3D-VLA: Lifting VLA Models to 3D Geometry and Dynamics-Aware Manipulation

2026-07-07

AI Summary

Existing vision-language-action (VLA) models struggle to jointly model 3D geometric structure and temporal action dynamics in evolving environments, limiting robotic precision in physical tasks. This work proposes Lift3D-VLA, a novel framework that geometrically aligns a pretrained 2D visual encoder into 3D point cloud space and introduces Geometry-Centric Masked Autoencoding (GC-MAE), a self-supervised mechanism that concurrently reconstructs current and predicts future geometric structures. Additionally, a hierarchical temporal action prediction module grounded in a large language model generates consistent and accurate action sequences. Experiments demonstrate that the method improves average success rates by 10.8% on MetaWorld and 11.1% on RLBench, outperforms the strongest baseline by 4 percentage points on real-world robotic tasks, and exhibits strong out-of-distribution generalization capabilities.

Abstract

Recently, Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse tasks. However, effective robotic manipulation in physical environments fundamentally requires geometric understanding and spatial reasoning. While some VLA approaches attempt to incorporate 3D information, they are constrained by limited data availability and geometric information loss in current 3D encoding pipelines, and fail to jointly capture 3D geometry and temporally structured actions in dynamic environments. To address these limitations, we introduce Lift3D-VLA, a unified VLA framework that equips models with explicit 3D point cloud reasoning and enables temporally coherent action generation. First, building upon our previous work Lift3D, an enhanced 2D model-lifting strategy is proposed to geometrically align 3D points with pretrained 2D positional embeddings. This design enables direct point-cloud encoding within the VLA vision encoder while minimizing spatial information loss. Based on explicit 3D inputs, we propose Geometry-Centric Masked Autoencoding (GC-MAE), a dual-objective self-supervised framework that reconstructs the current point cloud while predicting its future geometric evolution. This formulation allows the 2D vision encoder to internalize both 3D structure and physical dynamics. To fully exploit 3D representations, we further design layer-wise temporal action modeling, which leverages multiple layers of the LLM to collaboratively predict action chunks, enabling temporally consistent predictions. Across 22 simulated tasks and 8 real-world manipulation tasks, Lift3D-VLA achieves 10.8% and 11.1% higher mean success rates on MetaWorld and RLBench than the best-performing prior VLA methods, and outperforms the strongest real-world baseline by 4 percentage points, while exhibiting stronger generalization to out-of-distribution perturbations.

Problem

• 3D geometry
• temporal dynamics
• Vision-Language-Action models
• spatial reasoning
• robotic manipulation

Innovation

• 3D point cloud reasoning
• Geometry-Centric Masked Autoencoding
• temporal action modeling
• Vision-Language-Action models
• geometric alignment