Evaluation of Multilingual Ability to Use Spatial Deictic Expressions in Vision-Language Models

2026-07-08

AI Summary

This study addresses the challenge that current vision-language models struggle to accurately generate spatially grounded deictic expressions (e.g., “this,” “that”) in multilingual settings, largely due to insufficient modeling of language-specific spatial distinctions. To systematically evaluate this capability, the authors introduce the first benchmark encompassing four languages, leveraging controlled image–text pairings to assess how well models jointly reason about object distance and linguistic conventions when selecting appropriate demonstratives. The findings reveal that prevailing models significantly deviate from human-like behavior in both distance sensitivity and cross-linguistic expression patterns. By proposing a dedicated evaluation framework for multilingual spatial deixis, this work establishes a foundational methodology for a previously underexplored aspect of vision–language grounding.

Abstract

One of the expected abilities of vision-language models (VLMs) is spatial reasoning ability based on a given text and image. To evaluate the spatial reasoning abilities of VLMs, we focus on the use of spatial deictic expressions, which are defined as spatial expressions whose referent is determined by their situational context, such as ``this'' and ``that''. To handle spatial deictic expressions, VLMs must jointly reason over language and visual space, grounding context-dependent references in the image's spatial structure. In addition, selecting appropriate spatial deictic expressions across languages requires VLMs to understand the language-specific spatial distinctions encoded by these expressions. In this paper, we develop a benchmark to evaluate the multilingual ability of VLMs to use spatial deictic expressions in four languages. Our experiments using this benchmark reveal that the tested models use demonstratives in a manner different from that of humans, particularly in selecting the appropriate demonstratives based on the distance to the object.

Problem

• spatial deictic expressions
• vision-language models
• multilingual evaluation
• spatial reasoning
• demonstratives

Innovation

• spatial deictic expressions
• vision-language models
• multilingual evaluation
• spatial reasoning
• demonstratives