How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective

2025-09-23

AI Summary

Existing evaluation frameworks inadequately characterize visual spatial reasoning (VSR) capabilities of vision-language models (VLMs), and the conceptual boundaries of spatial intelligence remain ill-defined. Method: We propose the first three-tiered spatial intelligence taxonomy—perception → understanding → planning—and introduce SIBench, an open-source benchmark encompassing 23 diverse task scenarios. We conduct systematic, multimodal evaluations across architectures and training paradigms. Contribution/Results: Our analysis reveals that while current VLMs achieve reasonable performance on basic perceptual tasks, they exhibit substantial bottlenecks in spatial understanding (e.g., multi-view reasoning, temporal dynamic modeling) and spatial planning (e.g., numerical estimation, 3D reasoning). SIBench provides a standardized, empirically grounded tool for quantifying VSR capabilities and guiding targeted model improvements.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual Spatial Reasoning (VSR) is a core human cognitive ability and a critical requirement for advancing embodied intelligence and autonomous systems. Despite recent progress in Vision-Language Models (VLMs), achieving human-level VSR remains highly challenging due to the complexity of representing and reasoning over three-dimensional space. In this paper, we present a systematic investigation of VSR in VLMs, encompassing a review of existing methodologies across input modalities, model architectures, training strategies, and reasoning mechanisms. Furthermore, we categorize spatial intelligence into three levels of capability, ie, basic perception, spatial understanding, spatial planning, and curate SIBench, a spatial intelligence benchmark encompassing nearly 20 open-source datasets across 23 task settings. Experiments with state-of-the-art VLMs reveal a pronounced gap between perception and reasoning, as models show competence in basic perceptual tasks but consistently underperform in understanding and planning tasks, particularly in numerical estimation, multi-view reasoning, temporal dynamics, and spatial imagination. These findings underscore the substantial challenges that remain in achieving spatial intelligence, while providing both a systematic roadmap and a comprehensive benchmark to drive future research in the field. The related resources of this study are accessible at https://sibench.github.io/Awesome-Visual-Spatial-Reasoning/.

Problem

• Evaluating Vision-Language Models' visual spatial reasoning capabilities compared to humans
• Systematically investigating spatial intelligence across perception, understanding, and planning levels
• Benchmarking VLMs' performance gap between basic perception and complex spatial reasoning tasks

Innovation

• Systematically categorizes spatial intelligence capabilities
• Introduces SIBench benchmark with 20 datasets
• Reveals perception-reasoning gap in VLMs