Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

2025-05-22

AI Summary

Current multimodal large language models (MLLMs) exhibit limited spatial understanding, constrained to single-frame images and thus inadequate for real-world applications—such as robotics—that require multi-frame spatiotemporal reasoning. To address this, we propose MultiSPA, a novel framework enabling scalable and generalizable cross-frame spatial reasoning in 3D/4D scenes. Our approach integrates depth estimation, inter-frame visual correspondence, temporal dynamic modeling, and end-to-end MLLM fine-tuning. We introduce the first large-scale, multi-frame spatial understanding dataset (27M samples) and a unified evaluation benchmark. Experiments demonstrate that MultiSPA significantly outperforms both open- and closed-source baselines across diverse tasks; it exhibits emergent capabilities under challenging conditions—including occlusion and motion blur—and has been validated as an effective multi-frame reward annotator for robotic vision systems.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for robotics and other real-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with robust multi-frame spatial understanding by integrating depth perception, visual correspondence, and dynamic perception. Central to our approach is the MultiSPA dataset, a novel, large-scale collection of more than 27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable, generalizable multi-frame reasoning. We further observe multi-task benefits and early indications of emergent capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.

Problem

• Enhancing MLLMs for multi-frame spatial understanding in robotics
• Integrating depth, visual correspondence, and dynamic perception in MLLMs
• Addressing limitations of single-image spatial reasoning in real-world applications

Innovation

• Integrates depth, visual correspondence, dynamic perception
• Uses MultiSPA dataset with 27M 3D/4D samples
• Achieves scalable multi-frame reasoning for robotics