SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision

2026-06-18

AI Summary

Existing multimodal large language models (MLLMs) face a trade-off in acquiring 3D spatial awareness: they either rely on external tools, incurring high inference overhead, or employ implicit distillation, which lacks interpretability and geometric constraints. This work proposes a task-oriented visual supervision mechanism that, for the first time, enables explicit 2D-to-3D feature lifting within the MLLM itself, producing interpretable 3D representations such as depth maps, camera poses, and point clouds. By treating the 3D reconstruction process as a transparent diagnostic window into the model’s spatial understanding, the approach significantly enhances spatial reasoning performance across multiple benchmarks while supporting high-quality 3D reconstruction and semi-supervised generalization.

Abstract

Unlocking the spatial intelligence of multimodal large language model (MLLMs) is crucial for understanding and interacting with the 3D world. Prevailing approaches typically inject spatial priors via external tools, which impose significant inference overhead, or rely on latent feature distillation, which remains uninterpretable and lacks fine-grained geometric constraints. To address these issues, we propose SpatialSV, a framework designed to internalize robust 3D spatial awareness within MLLMs while simultaneously offering inherent interpretability. Deviating from passive feature imitation, SpatialSV employs task-oriented visual supervision, compelling the model to actively lift its 2D visual features into explicit 3D representations, including depth maps, camera poses, and point clouds. Crucially, this 2D-to-3D lifting process provides a transparent window into the model's representations: the resulting 3D reconstructions serve as an intuitive proxy for visualizing and diagnosing the quality of the model's intrinsic spatial knowledge. Extensive experiments across multiple models and benchmarks demonstrate the effectiveness of SpatialSV in enhancing and interpreting MLLMs' spatial intelligence. Furthermore, the framework exhibits strong generalization in semi-supervised settings, validating its potential to leverage unlabeled visual data for scalable, interpretable spatial representation learning.

Problem

• 3D spatial awareness
• multimodal large language models
• interpretable representation
• spatial intelligence
• visual supervision

Innovation

• 3D spatial awareness
• task-oriented visual supervision
• interpretable representation
• 2D-to-3D lifting
• multimodal large language models