G$^2$VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning

2025-11-26

AI Summary

Vision-language models (VLMs) exhibit limited robustness in spatial understanding and reasoning tasks, primarily due to their inability to implicitly recover 3D geometric structure from 2D images. Method: We propose the first unified framework that natively embeds 3D visual priors into VLMs, establishing a geometrically grounded architecture that jointly optimizes multi-view image/video-driven 3D reconstruction and contextual spatial reasoning. Our approach integrates explicit 3D geometric constraints, interleaved reasoning mechanisms, and in-context learning to unify 3D attribute prediction and spatial relation modeling. Contribution/Results: Experiments demonstrate that our model matches state-of-the-art feedforward methods on standard 3D reconstruction benchmarks while achieving SOTA or leading performance on multiple spatial reasoning benchmarks—including SQA and SpatialBench—significantly enhancing the generalization capability of VLMs for spatial intelligence.

Technology Category

• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Vision-Language Models (VLMs) still lack robustness in spatial intelligence, demonstrating poor performance on spatial understanding and reasoning tasks. We attribute this gap to the absence of a visual geometry learning process capable of reconstructing 3D space from 2D images. We present G$^2$VLM, a geometry grounded vision-language model that bridges two fundamental aspects of spatial intelligence: spatial 3D reconstruction and spatial understanding. G$^2$VLM natively leverages learned 3D visual geometry features to directly predict 3D attributes and enhance spatial reasoning tasks via in-context learning and interleaved reasoning. Our unified design is highly scalable for spatial understanding: it trains on abundant multi-view image and video data, while simultaneously leveraging the benefits of 3D visual priors that are typically only derived from hard-to-collect annotations. Experimental results demonstrate G$^2$VLM is proficient in both tasks, achieving comparable results to state-of-the-art feed-forward 3D reconstruction models and achieving better or competitive results across spatial understanding and reasoning tasks. By unifying a semantically strong VLM with low-level 3D vision tasks, we hope G$^2$VLM can serve as a strong baseline for the community and unlock more future applications, such as 3D scene editing.

Problem

• Addresses VLMs' poor spatial understanding and reasoning capabilities
• Integrates 3D reconstruction from 2D images with spatial reasoning tasks
• Enhances spatial intelligence using learned 3D geometry and in-context learning

Innovation

• Unified model for 3D reconstruction and spatial reasoning
• Learns 3D geometry from multi-view images and videos
• Uses in-context learning to enhance spatial understanding