Segment Any 3D Gaussians

2023-12-01

arXiv.org

AI Summary

This work addresses the lack of efficient promptable 3D segmentation methods for 3D Gaussian Splatting (3D-GS) representations. To this end, we propose the first real-time, promptable 3D segmentation framework. Methodologically, we introduce a scale-gated affinity feature embedding and a soft-scale gating mechanism; leverage knowledge distillation to transfer SAM’s 2D visual prompting capability into the 3D Gaussian space; and incorporate a scale-aware contrastive learning strategy to explicitly model multi-granularity segmentation ambiguity. Experiments demonstrate that our method achieves state-of-the-art accuracy while responding to 2D point or bounding-box prompts and completing high-fidelity 3D object segmentation in only 4 ms. This is the first work to enable millisecond-level promptable segmentation under the 3D-GS representation, establishing a new paradigm for neural rendering and interactive 3D understanding.

Abstract

This paper presents SAGA (Segment Any 3D GAussians), a highly efficient 3D promptable segmentation method based on 3D Gaussian Splatting (3D-GS). Given 2D visual prompts as input, SAGA can segment the corresponding 3D target represented by 3D Gaussians within 4 ms. This is achieved by attaching an scale-gated affinity feature to each 3D Gaussian to endow it a new property towards multi-granularity segmentation. Specifically, a scale-aware contrastive training strategy is proposed for the scale-gated affinity feature learning. It 1) distills the segmentation capability of the Segment Anything Model (SAM) from 2D masks into the affinity features and 2) employs a soft scale gate mechanism to deal with multi-granularity ambiguity in 3D segmentation through adjusting the magnitude of each feature channel according to a specified 3D physical scale. Evaluations demonstrate that SAGA achieves real-time multi-granularity segmentation with quality comparable to state-of-the-art methods. As one of the first methods addressing promptable segmentation in 3D-GS, the simplicity and effectiveness of SAGA pave the way for future advancements in this field. Our code will be released.

Problem

• Efficient 3D promptable segmentation
• Multi-granularity ambiguity in 3D
• Real-time segmentation in 3D-GS

Innovation

• 3D Gaussian Splatting based segmentation
• Scale-gated affinity feature learning
• Real-time multi-granularity 3D segmentation