Evaluating Foundation Models' 3D Understanding Through Multi-View Correspondence Analysis

2025-12-12

AI Summary

Existing benchmarks for evaluating the 3D understanding capability of foundational vision models lack no-fine-tuning, dense-feature-driven evaluation protocols. Method: We introduce the first fine-tuning-free benchmark for multi-view correspondence analysis, built upon MVImgNet. It establishes a novel image segmentation task with graded difficulty across novel viewpoints, featuring a first-of-its-kind four-level difficulty hierarchy driven by inter-view angular disparity. We extend the Hummingbird framework to 3D multi-view settings, integrating multi-view geometry, dense feature matching, and in-context evaluation. Contribution/Results: Our framework systematically evaluates eight state-of-the-art encoders (e.g., DINO, ViT-G). Experiments reveal that DINO-style models exhibit significantly superior robustness under large viewpoint shifts compared to 3D-aware models, which require dedicated multi-view adaptation. We open-source a fully reproducible evaluation toolkit, establishing a new paradigm for assessing the intrinsic 3D competence of pre-trained visual encoders.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Benchmarking 3D spatial understanding of foundation models is essential for real-world applications such as robotics and autonomous driving. Existing evaluations often rely on downstream finetuning with linear heads or task-specific decoders, making it difficult to isolate the intrinsic 3D reasoning ability of pretrained encoders. In this work, we introduce a novel benchmark for in-context 3D scene understanding that requires no finetuning and directly probes the quality of dense visual features. Building on the Hummingbird framework, which evaluates in-context 2D scene understanding, we extend the setup to the 3D Multi-View ImageNet (MVImgNet) dataset. Given a set of images from objects in specific angles (keys), we benchmark the performance of segmenting novel views (queries) and report the scores in 4 categories of easy, medium, hard, and extreme based on the key-query view contrast. We benchmark 8 state-of-the-art foundation models and show DINO-based encoders remain competitive across large viewpoint shifts, while 3D-aware models like VGGT require dedicated multi-view adjustments. Our code is publicly available at https://github.com/ToyeshC/open-hummingbird-3d-eval .

Problem

• Benchmarking 3D spatial understanding in foundation models
• Evaluating intrinsic 3D reasoning without finetuning
• Analyzing multi-view correspondence for real-world applications

Innovation

• Introduces a novel benchmark for in-context 3D scene understanding
• Extends the Hummingbird framework to 3D using Multi-View ImageNet dataset
• Evaluates models by segmenting novel views without any finetuning