Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models

2025-11-03

AI Summary

This work addresses two key limitations of multimodal large language models (MLLMs) in 3D spatial reasoning: weak cross-view consistency and insufficient spatiotemporal relationship modeling. To this end, we propose the Viewpoint Learning task and introduce Viewpoint-100K—the first large-scale dataset of 100K multi-view image pairs. Our method employs a two-stage training paradigm: (1) supervised fine-tuning (SFT) to inject foundational spatial knowledge, followed by (2) group-relative policy optimization (GRPO), a reinforcement learning framework that explicitly models object-centric multi-view image pairs. We further design a novel hybrid cold-start initialization strategy, substantially improving model generalization to viewpoint transformations and spatial relations. Experiments demonstrate significant performance gains on both in-domain and out-of-domain 3D reasoning benchmarks. The proposed approach establishes a transferable foundation for spatial cognition, with direct implications for robot navigation, autonomous driving, and 3D scene understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved 2D visual understanding, prompting interest in their application to complex 3D reasoning tasks. However, it remains unclear whether these models can effectively capture the detailed spatial information required for robust real-world performance, especially cross-view consistency, a key requirement for accurate 3D reasoning. Considering this issue, we introduce Viewpoint Learning, a task designed to evaluate and improve the spatial reasoning capabilities of MLLMs. We present the Viewpoint-100K dataset, consisting of 100K object-centric image pairs with diverse viewpoints and corresponding question-answer pairs. Our approach employs a two-stage fine-tuning strategy: first, foundational knowledge is injected to the baseline MLLM via Supervised Fine-Tuning (SFT) on Viewpoint-100K, resulting in significant improvements across multiple tasks; second, generalization is enhanced through Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm on a broader set of questions. Additionally, we introduce a hybrid cold-start initialization method designed to simultaneously learn viewpoint representations and maintain coherent reasoning thinking. Experimental results show that our approach significantly activates the spatial reasoning ability of MLLM, improving performance on both in-domain and out-of-domain reasoning tasks. Our findings highlight the value of developing foundational spatial skills in MLLMs, supporting future progress in robotics, autonomous systems, and 3D scene understanding.

Problem

• Evaluating MLLMs' ability to capture spatial information for 3D reasoning
• Improving cross-view consistency in multimodal large language models
• Activating spatial reasoning capabilities for robotics and autonomous systems

Innovation

• Two-stage fine-tuning strategy enhances spatial reasoning
• Hybrid cold-start initialization learns viewpoint representations
• Group Relative Policy Optimization algorithm boosts generalization