Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

2026-04-28

AI Summary

Existing zero-shot 3D visual grounding methods are hindered by imprecise semantics and coarse geometry of 3D candidate regions under open-vocabulary settings, as well as spatial redundancy in multi-view reasoning. This work proposes a novel paradigm based on multi-faceted 2D–3D consistency, formulating target disambiguation as a multiple-choice reasoning task for vision-language models through the synergistic integration of three modules: semantic alignment, instance refinement, and view distillation. By unifying large language models, vision-language models, cross-modal matching, back-projection geometric reconstruction, and camera-view clustering, the proposed method achieves 62.0% Acc@0.25 and 53.6% Acc@0.5 on ScanRefer, surpassing the previous state of the art by 6.4% and 4.0%, respectively—marking the first demonstration of efficient and robust zero-shot 3D grounding.

Abstract

Zero-shot 3D Visual Grounding (3DVG) is a critical capability for open-world embodied AI. However, existing methods are fundamentally bottlenecked by the poor quality of open-vocabulary 3D proposals, suffering from inaccurate categories and imprecise geometries, as well as the spatial redundancy of exhaustive multi-view reasoning. To address these challenges, we propose MCM-VG, a novel framework that achieves robust zero-shot 3DVG by explicitly establishing Multiple Consistent 2D-3D Mappings. Instead of passively relying on noisy 3D segments, MCM-VG enforces 2D-3D consistency across three fundamental dimensions to achieve precise target localization and reliable reasoning. First, a Semantic Alignment module corrects category mismatches via LLM-driven query parsing and coarse-to-fine 2D-3D matching. Second, an Instance Rectification module leverages VLM-guided 2D segmentations to reconstruct missing targets, back-projecting these reliable visual priors to establish accurate 3D geometries. Finally, to eliminate spatial redundancy, a Viewpoint Distillation module clusters 3D camera directions to extract optimal frames. By pairing these optimal RGB frames with Bird's Eye View maps into concise visual prompt sets, we formulate the final target disambiguation as a multiple-choice reasoning task for Vision-Language Models. Extensive evaluations on ScanRefer and Nr3D benchmarks demonstrate that MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding. Remarkably, it achieves 62.0\% and 53.6\% in Acc@0.25 and Acc@0.5 on ScanRefer, outperforming previous baselines by substantial margins of 6.4\% and 4.0\%.

Problem

• Zero-shot 3D Visual Grounding
• open-vocabulary 3D proposals
• spatial redundancy
• multi-view reasoning
• 3D candidate quality

Innovation

• Zero-shot 3D Visual Grounding
• 2D-3D Consistency
• Vision-Language Models
• Semantic Alignment
• Viewpoint Distillation