Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training

2026-08-20

AI Summary

本文提出一种新的框架，通过共蒸馏策略和双引导学习方案，提高开放词汇3D目标检测中新物体发现的可靠性和模型训练的鲁棒性。

Abstract

Recently, open-vocabulary 3D object detection (3D-OVD) has gained increasing attention for its ability to detect unseen objects in 3D scenes. Existing approaches typically adopt a two-stage pipeline that first discovers novel objects using foundation models and then trains a 3D-OVD model based on these discovered objects. Although effective, this pipeline often suffers from inaccurate localization and mismatched classification during the discovery stage, which subsequently limits the performance of the model training stage. To address these limitations, we advocate for improving both the reliability of novel object discovery and the robustness of model training, and propose an innovative framework. Specifically, for reliable discovery, our co-distillation strategy distills high-quality novel objects by applying Hungarian matching over a comprehensive score that incorporates geometric consistency, structural objectness, and semantic certainty. To enhance robust model training, we further propose a dual-guidance learning scheme, incorporating a scene-awareness-guided uncertainty regularization for the regression head and an LLM-guided hierarchical alignment for the classification head, effectively mitigating the negative effects of imprecise 3D bounding boxes and semantic ambiguity. Extensive experiments on SUN RGB-D and ScanNetV2 demonstrate that our method achieves significant performance gains over state-of-the-art approaches. Code is available at https://github.com/shangboyuan/Co-3DGT

Problem

• Open-Vocabulary 3D Object Detection
• Localization Inaccuracy
• Classification Mismatch

Innovation

• co-distillation
• dual-guidance learning
• open-vocabulary 3D object detection