Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

2026-03-24

AI Summary

Existing multimodal large language models lack explicit 3D geometric representations, limiting their ability to perform precise spatial understanding and reasoning. To address this, this work proposes an end-to-end 3D cognitive mapping framework that introduces, for the first time, an explicit 3D memory mechanism. By leveraging multi-view geometric modeling, the method anchors visual-semantic tokens into a unified 3D space, constructing a structured 3D cognitive map upon which the language model can directly conduct spatial reasoning. This approach effectively integrates semantic and geometric information, achieving state-of-the-art performance across multiple spatial reasoning benchmarks and significantly enhancing the model’s capacity for 3D scene understanding and question answering.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Precise spatial understanding from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs), as their visual representations are predominantly semantic and lack explicit geometric grounding. While existing approaches augment visual tokens with geometric cues from visual geometry models, their MLLM is still required to implicitly infer the underlying 3D structure of the scene from these augmented tokens, limiting its spatial reasoning capability. To address this issue, we introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images, where each token is grounded in 3D space and possesses both semantic and geometric information. By feeding these tokens into the MLLM, our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks. Code will be made publicly available.

Problem

• spatial understanding
• multimodal large language models
• 3D reasoning
• geometric grounding
• multi-view images

Innovation

• 3D cognitive map
• multi-view vision-language reasoning
• geometric grounding
• spatial reasoning
• multimodal large language models