Open-Vocabulary and Referring Segmentation for 3D Gaussians Using 2D Detectors

2026-06-29

AI Summary

This work addresses the limitations of existing language-driven semantic understanding methods for 3D Gaussian scenes, which rely on dense CLIP features and struggle with complex referring expressions while suffering from noise-sensitive instance grouping or requiring a predefined number of instances. To overcome these challenges, the paper introduces the first integration of a 2D open-vocabulary detector with 3D Gaussian Splatting, enabling each Gaussian point to learn instance-level semantic features. By leveraging multi-view rendering and semantic voting, the method constructs a View-Aggregated Semantic Distribution (VASD) that facilitates robust 3D instance segmentation and referring expression grounding. The approach achieves zero-shot generalization across expressions ranging from simple nouns to complex referring phrases, significantly outperforming prior methods on the LeRF-OVS, ScanNet, and Ref-LeRF benchmarks, with a 16.7% improvement in mIoU on zero-shot referring expression tasks.

Abstract

3D Gaussian Splatting (3DGS) has emerged at the forefront of 3D scene reconstruction. Extending 3DGS with language-driven, open-vocabulary understanding has gained significant attention for real-world applications such as embodied AI. Recent methods achieve this by learning an instance feature attribute and assigning semantics by distilling high-dimensional Contrastive Language-Image Pretraining (CLIP) features directly into the scene representation. However, the instance grouping mechanisms of these methods either require a predefined number of instances or suffer from noise in their bottom-up grouping strategies. Furthermore, the reliance on CLIP restricts semantic understanding to simple noun phrases, preventing complex spatial reasoning and referential expression grounding. We present GaussDet, a method that circumvents the need for dense CLIP features by leveraging discrete, open-vocabulary 2D object detectors with referring expression capabilities. We learn instance features for individual Gaussians to decompose the scene into 3D instance groups. By rendering these groups and aggregating semantic votes from multi-view 2D detections, we generate a robust View-Aggregated Semantic Label Distribution (VASD) for each 3D instance. This view-aggregation strategy acts as a strong regularizer, attenuating spurious labels caused by low-quality instance grouping. Our approach enables a straightforward, zero-shot extension from simple language queries to complex referential grounding. Extensive evaluations across two key tasks -- open-vocabulary segmentation (LeRF-OVS, ScanNet) and referring expression grounding (Ref-LeRF) -- demonstrate that GaussDet achieves consistent improvements over existing methods. Most notably, we achieve a substantial 16.7% mIoU improvement in referential grounding within a strict zero-shot setting.

Problem

• Open-Vocabulary Segmentation
• Referring Expression Grounding
• 3D Gaussian Splatting
• Zero-Shot Learning
• Semantic Scene Understanding

Innovation

• 3D Gaussian Splatting
• open-vocabulary segmentation
• referring expression grounding
• view-aggregated semantic labeling
• zero-shot learning