SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition

2024-01-31

AI Summary

To address the coarse and non-editable instance segmentation caused by ambiguous object boundaries in 3D Gaussian Splatting (3D-GS), this paper proposes a boundary-enhanced 3D interactive segmentation method. The approach tackles the problem by (1) introducing a novel Gaussian decomposition mechanism that precisely identifies and isolates sparse Gaussian ellipsoids representing object boundaries; (2) designing a training-free 2D foundation model-to-3D-GS space lifting framework, enabling lossless transfer of 2D segmentation priors—e.g., from SAM—to the 3D rendering manifold; and (3) integrating unsupervised boundary Gaussian detection with real-time rendering. Achieving millisecond-level inference, the method effectively eliminates segmentation aliasing artifacts, yielding high-fidelity, editable 3D instance masks. Experimental results demonstrate significant improvements in boundary accuracy and editability, providing robust support for downstream applications such as 3D scene editing and compositional manipulation.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Learning with Manifolds
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis, benefiting from its high-quality rendering results and real-time rendering speed. However, the 3D Gaussians learned by 3D-GS have ambiguous structures without any geometry constraints. This inherent issue in 3D-GS leads to a rough boundary when segmenting individual objects. To remedy these problems, we propose SAGD, a conceptually simple yet effective boundary-enhanced segmentation pipeline for 3D-GS to improve segmentation accuracy while preserving segmentation speed. Specifically, we introduce a Gaussian Decomposition scheme, which ingeniously utilizes the special structure of 3D Gaussian, finds out, and then decomposes the boundary Gaussians. Moreover, to achieve fast interactive 3D segmentation, we introduce a novel training-free pipeline by lifting a 2D foundation model to 3D-GS. Extensive experiments demonstrate that our approach achieves high-quality 3D segmentation without rough boundary issues, which can be easily applied to other scene editing tasks.

Problem

• 3D Gaussian Splattering
• Boundary Segmentation
• Image Clarity

Innovation

• SAGD method
• Gaussian decomposition technique
• 3D boundary segmentation