Geometric Action Model for Robot Policy Learning

2026-06-15

AI Summary

Existing robotic policies often rely on 2D images or their latent representations, which struggle to explicitly model 3D geometric relationships and thus limit performance in contact-intensive tasks. This work proposes the Geometric Action Model (GAM), which, for the first time, directly repurposes a pretrained Geometric Foundation Model (GFM) as a unified backbone for perception, prediction, and action decoding. With only lightweight architectural modifications, GAM enables language-conditioned 3D spatiotemporal world modeling and action generation. Through a co-designed perception encoder and causal future predictor—augmented with intermediate feature splitting and rerouting mechanisms—GAM substantially outperforms current large-scale baselines across diverse simulated and real-world robotic manipulation tasks, achieving notable advances in accuracy, robustness, inference speed, and model compactness.

Abstract

Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world. Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models, but they still operate primarily on 2D image frames or 2D-derived latent spaces, leaving implicit the 3D geometry required for contact-rich manipulation. We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy that directly repurposes a pretrained geometric foundation model (GFM) as a shared substrate for perception, temporal prediction, and action decoding. GAM splits the GFM at an intermediate layer: the shallow layers serve as an observation encoder, and a causal future predictor inserted at the split layer forecasts future latent tokens conditioned on language, proprioception, and action history. The predicted future tokens are then routed through the remaining GFM blocks for feature propagation and decoding, allowing a single backbone to produce both future geometry and actions. This design equips the GFM with language-conditioned temporal world modeling through minimal architectural modification while preserving its rich geometric priors. Across a broad suite of simulation and real-robot manipulation benchmarks, GAM is more accurate, more robust, faster, and lighter than current foundation-model-scale baselines.

Problem

• 3D geometry
• robot policy learning
• contact-rich manipulation
• vision-language-action models
• geometric reasoning

Innovation

• Geometric Action Model
• geometric foundation model
• language-conditioned policy
• 3D world modeling
• robot manipulation