Dual-Pathway Geometry-Aware MLLM for Spatial Intelligence

2026-05-24

AI Summary

Existing vision-language models struggle to simultaneously capture holistic 3D structural understanding and fine-grained metric estimation in spatial reasoning tasks, often relying on additional geometric inputs that incur high computational costs and limit generalization. This work proposes GAMSI, a model that operates solely on RGB images and internalizes two types of geometric priors within a unified autoregressive architecture. Its key innovations include Metric-Structure Decoupled Queries (MSDQ) and an Expert-Guided Visual Grounding module (EVG), augmented with dual-path attention and task-decoupling masks to enable efficient geometric extraction and alignment without external geometric cues. Trained via a two-stage curriculum on a newly curated multimodal spatial instruction fine-tuning dataset, MTS—comprising 13 distinct task categories—GAMSI achieves state-of-the-art performance across seven spatial intelligence benchmarks.

Abstract

Spatial understanding of the physical world from 2D visual inputs hinges on two complementary forms of geometric knowledge: holistic 3D structural perception and fine-grained metric scale estimation. Existing multimodal large language models (MLLMs) typically address only one facet, ingesting either depth maps or point clouds as additional model inputs, which incurs substantial computational overhead and inherits the generalization limitations of upstream prediction models. We propose GAMSI, a dual-pathway Geometry-Aware MLLM for Spatial Intelligence that takes only RGB images as input while internalizing both forms of geometric prior within a unified autoregressive backbone. Specifically, we introduce Metric-Structure Decoupled Queries (MSDQ) which employ two groups of learnable queries to respectively extract dense metric signals and sparse structural cues from the shared visual context, with a task-decoupled attention mask further preventing the two pathways from contaminating each other. Building on this, an Expert-Guided Visual Grounding (EVG) module projects the aggregated cues back to frame-level visual features and aligns them with vision foundation models, which serve purely as training-time supervision, rather than as model inputs. We further build a multi-task spatial instruction-tuning dataset (MTS) comprising 152{,}776 samples spanning 13 task types and three visual modalities, consolidated from six public datasets. Trained with a two-stage curriculum, GAMSI achieves state-of-the-art performance on seven spatial intelligence benchmarks.

Problem

• spatial intelligence
• geometric knowledge
• multimodal large language models
• 3D structural perception
• metric scale estimation

Innovation

• Geometry-Aware MLLM
• Metric-Structure Decoupled Queries
• Expert-Guided Visual Grounding
• Spatial Intelligence
• Dual-Pathway Architecture