Spatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning

2025-10-03

AI Summary

Vision-language models (VLMs) exhibit significant limitations in reasoning about spatial relationships—such as directionality, topology, and proximity—in 3D scenes and complex object layouts. Method: We propose SpatialViLT and its masked variant, MaskedSpatialViLT, a multi-task learning framework that explicitly incorporates geometric priors via depth maps, 3D coordinates, and edge maps. To enhance spatial semantics, we introduce SpatialEnsemble, an ensemble method that jointly optimizes multi-granularity spatial representations. Built upon vision-language pretraining architectures, the model jointly optimizes depth estimation, 3D coordinate regression, and edge-aware perception. Contribution/Results: Our approach achieves new state-of-the-art accuracy on the Visual Spatial Reasoning (VSR) benchmark. It is the first to systematically integrate structured spatial features—depth, geometry, and boundaries—into the VLM training paradigm, substantially improving recognition and logical reasoning over complex spatial configurations.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-language models (VLMs) have advanced multimodal reasoning but still face challenges in spatial reasoning for 3D scenes and complex object configurations. To address this, we introduce SpatialViLT, an enhanced VLM that integrates spatial features like depth maps, 3D coordinates, and edge maps through a multi-task learning framework. This approach enriches multimodal embeddings with spatial understanding. We propose two variants: SpatialViLT and MaskedSpatialViLT, focusing on full and masked object regions, respectively. Additionally, SpatialEnsemble combines both approaches, achieving state-of-the-art accuracy. Our models excel in spatial reasoning categories such as directional, topological, and proximity relations, as demonstrated on the challenging Visual Spatial Reasoning (VSR) dataset. This work represents a significant step in enhancing the spatial intelligence of AI systems, crucial for advanced multimodal understanding and real-world applications.

Problem

• Enhancing visual spatial reasoning in vision-language models
• Addressing challenges in 3D scenes and complex object configurations
• Improving directional, topological, and proximity relation understanding

Innovation

• Integrates spatial features via multi-task learning
• Proposes full and masked object region variants
• Combines approaches in ensemble for state-of-art accuracy