SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoning

2026-03-23

AI Summary

This work addresses the limitation of current large-scale pretrained vision encoders, which are predominantly trained on 2D images and thus struggle to capture 3D spatial relationships inherent in real-world scenes, thereby constraining their performance on downstream tasks. To overcome this, we propose SpatialBoost, a novel framework that, for the first time, leverages iterative language-guided chain-of-thought reasoning to transform dense 3D spatial information into structured textual descriptions and injects them into vision encoders such as DINOv3, enabling hierarchical spatial-aware representation learning. Evaluated on benchmarks requiring 3D scene understanding—including ADE20K—our method significantly enhances performance, improving DINOv3’s mIoU from 55.9 to 59.7 and achieving state-of-the-art results.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the remarkable success of large-scale pre-trained image representation models (i.e., vision encoders) across various vision tasks, they are predominantly trained on 2D image data and therefore often fail to capture 3D spatial relationships between objects and backgrounds in the real world, constraining their effectiveness in many downstream applications. To address this, we propose SpatialBoost, a scalable framework that enhances the spatial awareness of existing pre-trained vision encoders by injecting 3D spatial knowledge expressed in linguistic descriptions. The core idea involves converting dense 3D spatial information from 2D images into linguistic expressions, which is then used to inject such spatial knowledge into vision encoders through a Large Language Model (LLM). To this end, we adopt a multi-turn Chain-of-Thought (CoT) reasoning process that progressively incorporates dense spatial knowledge and builds hierarchical spatial understanding. To validate effectiveness, we adapt SpatialBoost to state-of-the-art vision encoders such as DINOv3, and evaluate its performance gains on a wide range of benchmarks requiring both 3D perception and general vision abilities. For instance, SpatialBoost improves DINOv3 performance from 55.9 to 59.7 mIoU on ADE20K, achieving state-of-the-art performance with 3.8% gain over the pre-trained DINOv3.

Problem

• 3D spatial relationships
• vision encoders
• spatial awareness
• 2D image data
• downstream applications

Innovation

• SpatialBoost
• language-guided reasoning
• 3D spatial awareness
• Chain-of-Thought
• vision-language integration