Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting

2025-03-18

AI Summary

This paper addresses key challenges—such as correspondence errors and cumbersome pre-/post-processing—in lifting multi-view 2D instance segmentation to 3D Gaussian splatting. To this end, we propose Unified-Lift, an end-to-end framework. Its core contributions are: (1) embedding instance-aware features into the 3D Gaussian point cloud and introducing a learnable, object-level semantic codebook for explicit object-level semantic modeling; (2) pioneering Gaussian-point-level contrastive learning to enhance feature discriminability, jointly with an association learning module and a noise-label filtering mechanism to mitigate codebook degradation; and (3) enabling robust training under noisy supervision. Unified-Lift achieves a +12.6% mAP improvement over prior methods on LERF-Masked, Replica, and Messy Rooms benchmarks, while accelerating inference by 2.3×. The source code is publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Lifting multi-view 2D instance segmentation to a radiance field has proven to be effective to enhance 3D understanding. Existing methods rely on direct matching for end-to-end lifting, yielding inferior results; or employ a two-stage solution constrained by complex pre- or post-processing. In this work, we design a new end-to-end object-aware lifting approach, named Unified-Lift that provides accurate 3D segmentation based on the 3D Gaussian representation. To start, we augment each Gaussian point with an additional Gaussian-level feature learned using a contrastive loss to encode instance information. Importantly, we introduce a learnable object-level codebook to account for individual objects in the scene for an explicit object-level understanding and associate the encoded object-level features with the Gaussian-level point features for segmentation predictions. While promising, achieving effective codebook learning is non-trivial and a naive solution leads to degraded performance. Therefore, we formulate the association learning module and the noisy label filtering module for effective and robust codebook learning. We conduct experiments on three benchmarks: LERF-Masked, Replica, and Messy Rooms datasets. Both qualitative and quantitative results manifest that our Unified-Lift clearly outperforms existing methods in terms of segmentation quality and time efficiency. The code is publicly available at href{https://github.com/Runsong123/Unified-Lift}{https://github.com/Runsong123/Unified-Lift}.

Problem

• Improves 3D scene segmentation from 2D multi-view images.
• Introduces a unified end-to-end approach for accurate 3D segmentation.
• Addresses challenges in codebook learning for object-level understanding.

Innovation

• Unified-Lift: end-to-end object-aware lifting approach
• Learnable object-level codebook for explicit understanding
• Association and noisy label filtering for robust learning