Segment then Splat: A Unified Approach for 3D Open-Vocabulary Segmentation based on Gaussian Splatting

2025-03-28

AI Summary

Existing methods predominantly rely on 2D pixel-wise parsing, leading to multi-view inconsistency, low accuracy in 3D object retrieval, and poor generalization to dynamic scenes. To address these limitations, we propose the first open-vocabulary, true 3D semantic segmentation framework for both static and dynamic 3D scenes—breaking the conventional “reconstruct-then-segment” paradigm. Our method introduces a novel “segment-first-then-reconstruct” pipeline: it performs object-pre-grouped 3D Gaussian ellipsoid segmentation *before* scene reconstruction, achieving segmentation-by-construction; resolves Gaussian-object misalignment in dynamic scenes; and bypasses explicit language field learning by directly leveraging CLIP’s text-image embeddings for zero-shot, open-vocabulary querying. Extensive evaluation across multiple datasets demonstrates superior cross-view consistency and high-precision 3D object retrieval for both static and dynamic scenarios.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment then Splat reverses the long established approach of"segmentation after reconstruction"by dividing Gaussians into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This approach not only eliminates Gaussian-object misalignment issues in dynamic scenes but also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.

Problem

• Addresses 3D open-vocabulary segmentation inconsistencies in multi-view 2D methods
• Solves dynamic scene challenges in 3D segmentation via Gaussian Splatting
• Eliminates Gaussian-object misalignment and accelerates optimization in segmentation

Innovation

• Segment before reconstruction using Gaussian Splatting
• Assign CLIP embeddings for open-vocabulary querying
• Unified approach for static and dynamic scenes