Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models

2026-09-07

AI Summary

研究通过构建POVBench数据集，评估了视觉-语言模型在理解基于上下文的观察者视角和解释空间关系方面的能力。

Abstract

Reasoning over language instructions in embodied tasks such as robotics often requires understanding spatial relations from a speaker's situated perspective. Humans infer such perspectives from shared environmental knowledge, activity context, and commonsense. Recent vision-language models (VLMs) appear capable of spatial reasoning, but their ability to infer a speaker's viewpoint from contextual cues and interpret situated spatial relations from that viewpoint remains unclear. We call this capability contextual observer grounding. To study this capability, we construct the Point-of-View Benchmark (POVBench), a dataset of 3D scenes and queries that disentangles Inferred, Stated, and Given forms of observer grounding in natural embodied communication. Given multi-view observations and a natural-language sentence, models must localize unseen or underspecified targets from situated spatial and contextual cues. Across state-of-the-art VLMs, localizing targets from directional language remains challenging, even when observer grounding is made explicit. We find that explicit breakdowns of observer-relative spatial reasoning improve target localization. Our project page is available at https://mimo-owl.github.io/POVBench/.

Problem

• Contextual Observer Grounding
• Vision-Language Models
• Spatial Reasoning
• Embodied Tasks

Innovation

• Contextual Observer Grounding
• POVBench
• Spatial Reasoning
• Vision-Language Models
• Embodied Communication