When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning

2026-07-13

AI Summary

This work addresses the challenge that vision-language models struggle to accurately reason about depth and relative spatial positions among objects in 3D space. To this end, the authors propose Depth Ordinal Prompting (DOP), a training-free method that converts object-level depth orderings—derived from off-the-shelf monocular depth estimators—into concise, directional natural language ordinal cues, which are directly embedded into the model’s input prompt. The study demonstrates that incorporating depth information in textual form outperforms feeding raw depth maps and does not degrade performance when the original image already contains sufficient spatial cues. Extensive experiments show that DOP consistently enhances spatial reasoning across diverse benchmarks, models, and depth estimators, achieving superior or comparable results to existing training-free depth prompting approaches, particularly when depth signals are reliable.

Abstract

Vision-language models (VLMs) are expected to reason about physical space -- which object is closer, what lies behind what, and how objects are arranged in 3D -- yet they still struggle with such spatial judgments. A natural remedy is to show the model a depth map, but we find that this can make performance worse. We show that depth is not absent: it reaches the language model, but becomes difficult to access for downstream reasoning, while rendered pseudo-depth maps act as noisy auxiliary images that frozen VLMs cannot easily regulate. We propose Depth-Ordinal Prompting (DOP), a training-free method that converts monocular depth into a single question-targeted ordinal text cue at the queried objects, without adding a depth image, training a module, injecting features, or using labels. Our key finding is form dependence: the same depth signal can hurt when shown as an image but help when told as text.Across benchmarks, models, and depth estimators, DOP improves spatial reasoning when pseudo-depth provides reliable object-level ordering and remains largely neutral in strong original-image regimes. It is also competitive with the strongest training-free depth-prompting alternative while being simpler and more targeted.

Problem

• spatial reasoning
• vision-language models
• depth perception
• monocular depth
• 3D scene understanding

Innovation

• Depth-Ordinal Prompting
• vision-language models
• spatial reasoning
• monocular depth
• training-free method