DM-KG: A Novel Method for Boosting Spatial Cognition of Vision-Language Models in Street View Imagery

2026-07-13

AI Summary

This work addresses the issue of spatial-semantic hallucination in vision-language models when processing street-view images by introducing the DM-KG framework, which, for the first time, injects a directional-metric knowledge graph as geometric prior to enhance spatial reasoning. The method leverages panoptic segmentation and metric depth estimation to derive entity-level 3D coordinates, encoding pairwise clock-face azimuths and Euclidean distances between entities into a structured knowledge graph. Experimental results on public spatial question-answering benchmarks demonstrate that the proposed approach reduces the mean absolute error in distance estimation by 31.1% and decreases the mean angular error in direction judgment by 65.8%, while maintaining high question-answering success rates, thereby significantly improving both the accuracy and interpretability of spatial reasoning.

Abstract

As vision-language models (VLMs) are increasingly deployed in geospatial question answering and visual scene understanding, improving their spatial cognition capability on street view imagery for complex logical reasoning has emerged as a key research priority. However, existing VLMs frequently suffer from "spatial semantic hallucinations" when perceiving object locations, distances, and directions in real-world street view scenes. Furthermore, such errors are often recalcitrant to tracing and calibration, posing a critical bottleneck for their practical deployment in geospatial tasks. To address this pressing challenge, this study proposes DM-KG (Direction-Metric Knowledge Graph), a structurally grounded spatial representation framework for street view imagery. By explicitly extracting directional and metric relationships between entities from a single 2D image, this framework enhances the spatial reasoning accuracy of VLMs through a structured knowledge graph. Specifically, we integrate panoptic segmentation with metric depth estimation to robustly compute entity-level 3D spatial coordinates. Subsequently, we encode the clock azimuths and Euclidean distances of entity pairs into a JSON-formatted knowledge graph, which is injected into the VLM as an explicit geometric prior to guide spatial reasoning. Experimental results on public spatial question-answering (QA) benchmarks demonstrate that DM-KG reduces the mean absolute error (MAE) in distance estimation by 31.1% and the mean angular error in direction judgment by 65.8%, while simultaneously maintaining a high QA success rate. By establishing a complete, augmented reasoning pipeline, this research significantly improves the spatial cognitive capabilities of VLMs in street view scenarios, thereby providing a flexible, generalized, and interpretable framework for geographic visual question answering (GeoVQA) in open environments.

Problem

• spatial cognition
• vision-language models
• street view imagery
• spatial semantic hallucinations
• geospatial question answering

Innovation

• Direction-Metric Knowledge Graph
• Spatial Cognition
• Vision-Language Models
• Geospatial Reasoning
• Street View Imagery