Mind over Space: Can Multimodal Large Language Models Mentally Navigate?

2026-03-23

AI Summary

Existing multimodal large language models lack human-like spatial reasoning and mental navigation capabilities over long spatiotemporal scales, hindering their ability to construct cognitive maps from experience and perform prospective path planning. To address this limitation, this work introduces Video2Mental, a novel benchmark that incorporates the cognitive science concept of “mental navigation” into AI evaluation for the first time, and proposes NavMind—a model that bridges perception and structured planning through learnable, fine-grained internalized cognitive maps. Leveraging progressive difficulty-based supervised fine-tuning, explicit cognitive map modeling, multimodal video understanding, and stepwise path reasoning, NavMind substantially outperforms state-of-the-art commercial and spatially specialized models on long-horizon mental navigation tasks, demonstrating superior planning accuracy and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the widespread adoption of MLLMs in embodied agents, their capabilities remain largely confined to reactive planning from immediate observations, consistently failing in spatial reasoning across extensive spatiotemporal scales. Cognitive science reveals that Biological Intelligence (BI) thrives on "mental navigation": the strategic construction of spatial representations from experience and the subsequent mental simulation of paths prior to action. To bridge the gap between AI and BI, we introduce Video2Mental, a pioneering benchmark for evaluating the mental navigation capabilities of MLLMs. The task requires constructing hierarchical cognitive maps from long egocentric videos and generating landmark-based path plans step by step, with planning accuracy verified through simulator-based physical interaction. Our benchmarking results reveal that mental navigation capability does not naturally emerge from standard pre-training. Frontier MLLMs struggle profoundly with zero-shot structured spatial representation, and their planning accuracy decays precipitously over extended horizons. To overcome this, we propose \textbf{NavMind}, a reasoning model that internalizes mental navigation using explicit, fine-grained cognitive maps as learnable intermediate representations. Through a difficulty-stratified progressive supervised fine-tuning paradigm, NavMind effectively bridges the gap between raw perception and structured planning. Experiments demonstrate that NavMind achieves superior mental navigation capabilities, significantly outperforming frontier commercial and spatial MLLMs.

Problem

• mental navigation
• spatial reasoning
• multimodal large language models
• cognitive maps
• embodied agents

Innovation

• mental navigation
• cognitive maps
• multimodal large language models
• spatial reasoning
• NavMind