VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation

2026-07-14

AI Summary

Existing vision-language-action (VLA) models lack explicit scene-level 3D representations, hindering their ability to reason effectively about spatial layouts and geometric constraints. This work proposes a geometry- and semantics-integrated 3D cognitive representation that lifts multi-view vision-language features into 3D space using 3D Gaussian primitives. A novel Merge-then-Query mechanism compresses these features into compact context tokens, achieving a 99% token compression rate while preserving action-relevant 3D structural information. Evaluated on seven real-world tasks, the method improves task success rates by 22.8% over baseline VLA models and demonstrates a 30.0% gain on out-of-distribution challenge tasks compared to VLA-Adapter, significantly enhancing spatial reasoning and generalization capabilities in VLA systems.

Abstract

Vision-Language-Action (VLA) models have emerged as a powerful end-to-end paradigm for robotic manipulation by mapping language instructions and 2D visual inputs directly to actions. However, these models lack an explicit, scene-level 3D representation, limiting their ability to reason over spatial layouts and geometric constraints. While recent efforts incorporate explicit 3D cues, such as depth maps or point clouds, to improve geometric awareness, they primarily capture low-level structures and lack high-level semantic grounding in 3D space. In human cognition, interaction with the physical world relies on a 3D semantic cognitive map - an internal mental model that integrates spatial layouts with semantic context to enable persistent, viewpoint-invariant reasoning. In light of this, we present VistaVLA, a novel two-stage framework that constructs a geometry- and semantics-aware 3D cognitive representation from 3D Gaussian primitives and grounds it as compact context tokens for VLA policy learning. Specifically, VistaVLA lifts multi-view vision-language features into 3D Gaussian primitives, forming geometry-anchored semantic tokens that align view-consistent spatial grounding with 2D visual feature spaces. To make this 3D representation computationally tractable for effective VLA control, we introduce Merge-then-Query (MtQ), a token summarization mechanism. MtQ compresses dense Gaussian primitives into a highly compact set of spatially informative tokens, achieving a 99% token reduction while preserving action-relevant 3D layouts and semantic context. Extensive evaluations in both simulated and real-world environments demonstrate the effectiveness of VistaVLA. Notably, in real-world scenarios, VistaVLA improves success rates by 22.8% across seven real-world tasks and by 30.0% over the VLA-Adapter baseline on challenging out-of-distribution tasks.

Problem

• Vision-Language-Action
• 3D representation
• geometric reasoning
• semantic grounding
• robotic manipulation

Innovation

• 3D Gaussian Splatting
• Vision-Language-Action (VLA)
• Semantic Cognitive Map
• Merge-then-Query
• Geometric Reasoning