Robotic Manipulation is Vision-to-Geometry Mapping ($f(v) \rightarrow G$): Vision-Geometry Backbones over Language and Video Models

2026-04-14

AI Summary

Existing robotic manipulation approaches rely on vision-language or video models whose representations are constrained by semantic priors or 2D assumptions, limiting their ability to accurately capture the 3D geometric structures essential for physical interaction. This work proposes the Vision-Geometry-Action (VGA) model, which, for the first time, leverages a pretrained 3D world model as its backbone to directly map visual inputs to geometric actions, bypassing indirect modeling pathways based on language or 2D videos. By incorporating a progressive voxel modulation module and a joint training strategy, VGA outperforms state-of-the-art baselines such as π₀.₅ and GeoVLA in simulation and demonstrates exceptional zero-shot viewpoint generalization in real-world settings, significantly surpassing current methods.

Technology Category

• Computer Vision: Large Vision Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

At its core, robotic manipulation is a problem of vision-to-geometry mapping ($f(v) \rightarrow G$). Physical actions are fundamentally defined by geometric properties like 3D positions and spatial relationships. Consequently, we argue that the foundation for generalizable robotic control should be a vision-geometry backbone, rather than the widely adopted vision-language or video models. Conventional VLA and video-predictive models rely on backbones pretrained on large-scale 2D image-text or temporal pixel data. While effective, their representations are largely shaped by semantic concepts or 2D priors, which do not intrinsically align with the precise 3D geometric nature required for physical manipulation. Driven by this insight, we propose the Vision-Geometry-Action (VGA) model, which directly conditions action generation on pretrained native 3D representations. Specifically, VGA replaces conventional language or video backbones with a pretrained 3D world model, establishing a seamless vision-to-geometry mapping that translates visual inputs directly into physical actions. To further enhance geometric consistency, we introduce a Progressive Volumetric Modulation module and adopt a joint training strategy. Extensive experiments validate the effectiveness of our approach. In simulation benchmarks, VGA outperforms top-tier VLA baselines including $π_{0.5}$ and GeoVLA, demonstrating its superiority in precise manipulation. More importantly, VGA exhibits remarkable zero-shot generalization to unseen viewpoints in real-world deployments, consistently outperforming $π_{0.5}$. These results highlight that operating on native 3D representations-rather than translating through language or 2D video priors-is a highly promising direction for achieving generalizable physical intelligence.

Problem

• Robotic Manipulation
• Vision-to-Geometry Mapping
• 3D Representation
• Generalizable Control
• Geometric Consistency

Innovation

• vision-to-geometry mapping
• 3D world model
• robotic manipulation
• geometric consistency
• zero-shot generalization