MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs

2026-07-09

AI Summary

Current vision-language models (VLMs) lack the capacity to integrate multi-view information to construct allocentric three-dimensional mental models, and no effective evaluation framework exists for this capability. To address this gap, this work introduces MultiView-Bench, a diagnostic benchmark that uniquely focuses on assessing VLMs’ ability to decouple egocentric viewpoints and reason about cross-view 3D spatial relationships within a globally anchored coordinate system. We further propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints and fuses multi-view evidence, substantially enhancing the performance of diverse baseline models—by up to 3–5×—under strict observation budgets. Our findings reveal systematic deficiencies in existing VLMs regarding 3D scene understanding and provide a principled pathway toward meaningful improvement.

Abstract

Recent benchmarks for VLMs largely assess single- or limited-view perception, leaving untested the core cognitive ability to integrate observations across viewpoints into a coherent, world-centric (allocentric) 3D mental model. We introduce MultiView-Bench, a diagnostic benchmark expressly designed to evaluate multi-view integration for holistic 3D scene comprehension. Unlike existing datasets that focus on pixel-level mapping or camera-relative navigation, MultiView-Bench requires models to decouple object positioning from transient perspectives and ground them in a fixed global coordinate system. This capability serves as a prerequisite for VLMs before being deployed for downstream tasks such as mechanical part assembly. Our systematic evaluation of frontier VLMs reveals consistent failure modes: strong performance on 2D planar relations from a single image, but marked difficulty with 3D spatial relations and with aggregating information across views. We further identify biases in VLMs, such as struggles with unconventional axis directions and sensitivity to object colorways and texture variations. Acknowledging these limitations, we propose ViewNavigator, a multi-agent framework that actively selects informative viewpoints, perceives, and fuses multi-view evidence, improving diverse base models on MultiView-Bench even under a strict budget-matched comparison (and by 3-5x for the full agent).

Problem

• multi-view integration
• world-centric representation
• 3D scene comprehension
• visual language models
• spatial reasoning

Innovation

• multi-view integration
• world-centric reasoning
• 3D scene understanding
• visual language models
• active viewpoint selection