GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models

2026-03-09

AI Summary

Existing vision-language-action (VLA) models lack explicit modeling of 3D geometric structure, limiting their capacity for high-precision spatial reasoning and manipulation. To address this, this work proposes the Gaussian Spatial Tokenizer (GST), which converts depth and semantic features into geometrically meaningful 3D Gaussian primitives. It further introduces a 3D depth-aware chain-of-thought (DA-CoT) mechanism to enable multi-stage, structured spatial reasoning. This approach is the first in VLA to explicitly represent local surface orientation and geometric confidence, integrating spatial attention pooling, a mixture-of-experts flow-matching decoder, and conditional ODE integration. The method achieves state-of-the-art performance on LIBERO and SimplerEnv, with success rates of 96.4% (+2.0%) and 80.2% (+5.4%), respectively, significantly advancing fine-grained robotic manipulation capabilities.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure. We introduce GST-VLA with two contributions. First, the Gaussian Spatial Tokenizer (GST) converts frozen dense depth and frozen semantic patch features into $N_g{=}128$ anisotropic 3D Gaussian primitives, each parameterized by a metric residual mean $μ\in \mathbb{R}^3$, log-scale covariance $\log σ\in \mathbb{R}^3$, and learned opacity $α\in (0,1)$. The covariance eigenstructure encodes local surface orientation, and opacity provides per-primitive geometric confidence, both inaccessible from scalar depth. Spatial attention pooling with learned queries concentrates the fixed token budget on geometrically salient regions rather than distributing uniformly. Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts, covering 3D object grounding, grasp affordance contact geometry, pairwise metric distances, and coarse SE(3) waypoints, as explicit generation targets in the training loss. A cross-attention sublayer at every VLM transformer block provides direct access to the raw 256-primitive Gaussian field during DA-CoT generation. A 300M-parameter flow-matching action expert with mixture-of-experts feedforward sublayers decodes 7-DoF delta action chunks via conditional ODE integration, conditioned on both VLM hidden states and DA-CoT outputs through dual cross-attention. Trained with composite $\mathcal{L}_\mathrm{flow} + \mathcal{L}_\mathrm{CoT} + \mathcal{L}_\mathrm{depth}$ across three progressive stages, GST-VLA achieves 96.4% on LIBERO (+2.0%), and 80.2% on SimplerEnv (+5.4%). Ablations isolate the contribution of each GST component, each DA-CoT thought, and each training stage, confirming independent and synergistic gains concentrated on precision demanding tasks.

Problem

• Vision-Language-Action
• 3D Geometry
• Depth Awareness
• Spatial Reasoning
• Robot Manipulation

Innovation

• Gaussian Spatial Tokenizer
• 3D Depth-Aware Chain-of-Thought
• Vision-Language-Action Models
• Anisotropic 3D Gaussians
• Flow-Matching Action Decoding