Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding

2026-09-04

AI Summary

该研究通过训练一个轻量级视图选择器来解决3D视觉定位中视角选择的问题，从而提高基于自然语言查询的3D物体定位准确性。

Abstract

Recent zero-shot 3D visual grounding methods leverage vision-language models (VLMs) to localize objects in 3D scenes from natural language queries. However, these methods typically rely on heuristic rules to select which camera views are provided to the VLM, often prioritizing object visibility rather than grounding relevance. We present IVSGround, a framework that learns Influential View Selection for VLM-based 3D visual grounding. Instead of using fixed heuristics, a lightweight view selector is trained to identify views that provide discriminative evidence for grounding. To obtain supervision signals, we generate training signals using feedback from a reasoning VLM through a two-stage rejection sampling process. During inference, the learned selector predicts query-conditioned influential views for each candidate object, which are then evaluated by a frozen reasoning VLM through comparative grounding. Experiments on ScanRefer and NR3D show that IVSGround consistently improves grounding accuracy over existing zero-shot pipelines, demonstrating that selecting where to look is crucial for effective 3D visual grounding. Project page: https://ivsground.github.io/

Problem

• 3D visual grounding
• vision-language models
• view selection
• zero-shot

Innovation

• Influential View Selection
• Vision-Language Models
• 3D Visual Grounding
• Zero-shot