SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models

2025-05-01

AI Summary

Current large multimodal models (LMMs) suffer from strong 2D biases and insufficient 3D training data, resulting in severely limited 3D spatial reasoning capabilities. To address this, we propose SpatialLLM—the first systematic framework for enhancing 3D spatial understanding in LMMs. Methodologically, we introduce the first VQA dataset integrating real-world images with explicit 3D orientation relations; design a joint data-architecture-training optimization paradigm comprising 3D-aware probing, dialogue-based data construction, multi-stage fine-tuning, and a plug-and-play spatial relation modeling module. Experiments demonstrate that SpatialLLM outperforms GPT-4o by 8.7% on dedicated 3D spatial reasoning benchmarks and significantly improves geometric understanding and reasoning in complex scenarios such as vehicle collision prediction. Our work establishes a novel paradigm for multimodal 3D cognitive modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Humans naturally understand 3D spatial relationships, enabling complex reasoning like predicting collisions of vehicles from different directions. Current large multimodal models (LMMs), however, lack of this capability of 3D spatial reasoning. This limitation stems from the scarcity of 3D training data and the bias in current model designs toward 2D data. In this paper, we systematically study the impact of 3D-informed data, architecture, and training setups, introducing SpatialLLM, a large multi-modal model with advanced 3D spatial reasoning abilities. To address data limitations, we develop two types of 3D-informed training datasets: (1) 3D-informed probing data focused on object's 3D location and orientation, and (2) 3D-informed conversation data for complex spatial relationships. Notably, we are the first to curate VQA data that incorporate 3D orientation relationships on real images. Furthermore, we systematically integrate these two types of training data with the architectural and training designs of LMMs, providing a roadmap for optimal design aimed at achieving superior 3D reasoning capabilities. Our SpatialLLM advances machines toward highly capable 3D-informed reasoning, surpassing GPT-4o performance by 8.7%. Our systematic empirical design and the resulting findings offer valuable insights for future research in this direction.

Problem

• Enhancing 3D spatial reasoning in large multimodal models
• Addressing scarcity of 3D training data for LMMs
• Improving 3D-informed architecture and training designs

Innovation

• Develops 3D-informed probing and conversation datasets
• Integrates 3D data with LMM architecture and training
• Achieves superior 3D reasoning surpassing GPT-4o by 8.7%