Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task

2026-07-01

AI Summary

This work addresses the challenge that current vision-language models (VLMs) struggle to disentangle visual depth cues from linguistic biases in depth perception. The authors propose the first evaluation framework explicitly designed to decouple these influences by constructing controlled 2D images derived from both real and synthetic 3D scenes, enabling isolated manipulation of individual depth cues. They introduce the O3-D dataset, comprising 37K images and 147K question-answer pairs, grounded in psychophysical tasks of depth ordering and “odd-one-out” detection. Additionally, they define novel metrics to quantify model sensitivity to visual versus linguistic signals and evaluate VLMs using chain-of-thought (CoT) and in-context learning (ICL) prompting. Experiments reveal that 12 state-of-the-art VLMs achieve only 47%–56% accuracy on depth ordering—barely above chance—and exhibit substantial interference from language biases.

Abstract

In this paper, we study depth perception of vision-language models (VLMs) to isolate the effects of pictorial depth cues and disentangle vision and language influences on model performance. To this end, we combine depth-ordering and odd-one-out psychophysical tasks: the VLMs are presented with images where one object is at different depth relative to other, otherwise identical, objects, and must determine whether the odd-one-out target is closer or farther to the observer. To create stimuli, we generate 2D views from simulated and real 3D scenes while controlling the presence of individual pictorial depth cues, enabling a fine-grained analysis of cue-level contributions. Language effects are examined by varying referring expression clarity. We also introduce a novel metric to quantify vision-vs-language sensitivities. Applying this methodology, we create the Odd-One-Out Depth (O3-D) dataset with 37K real and synthetic images and 147K image-question pairs. Evaluation of 12 open-source and commercial models on O3-D shows under-utilization of depth cues and depth-ordering accuracies between 47% and 56%, with no model above chance level. At the same time, our metric reveals strong linguistic bias in the answers. Neither chain-of-thought (CoT) nor in-context learning (ICL) significantly improves performance, suggesting that static image data alone may be insufficient for depth understanding. All code, the image generation pipeline, and the O3-D dataset are publicly released at https://github.com/lyiqian/o3-d.

Problem

• depth perception
• vision-language models
• pictorial cues
• language bias
• depth ordering

Innovation

• disentanglement
• pictorial depth cues
• vision-language models
• depth perception
• linguistic bias