PanopticSplatting: End-to-End Panoptic Gaussian Splatting

2025-03-23

AI Summary

To address the disconnection between geometric reconstruction and semantic/instance understanding, error accumulation from multi-stage pipelines, and reliance on hand-crafted modules in open-vocabulary panoptic 3D reconstruction, this paper proposes the first end-to-end differentiable Gaussian splatting framework. Our method jointly optimizes geometry and semantics via (1) a query-guided Gaussian segmentation mechanism coupled with frustum-local cross-attention, and (2) label mixing and 2D prediction warping to suppress pseudo-label noise and enhance cross-view consistency. Fully eliminating multi-stage design and manual priors, our approach achieves state-of-the-art panoptic reconstruction performance on ScanNet-V2 and ScanNet++, significantly outperforming NeRF-based and existing Gaussian-based methods. Moreover, it natively supports integration with diverse Gaussian foundation models.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance

Abstract

Open-vocabulary panoptic reconstruction is a challenging task for simultaneous scene reconstruction and understanding. Recently, methods have been proposed for 3D scene understanding based on Gaussian splatting. However, these methods are multi-staged, suffering from the accumulated errors and the dependence of hand-designed components. To streamline the pipeline and achieve global optimization, we propose PanopticSplatting, an end-to-end system for open-vocabulary panoptic reconstruction. Our method introduces query-guided Gaussian segmentation with local cross attention, lifting 2D instance masks without cross-frame association in an end-to-end way. The local cross attention within view frustum effectively reduces the training memory, making our model more accessible to large scenes with more Gaussians and objects. In addition, to address the challenge of noisy labels in 2D pseudo masks, we propose label blending to promote consistent 3D segmentation with less noisy floaters, as well as label warping on 2D predictions which enhances multi-view coherence and segmentation accuracy. Our method demonstrates strong performances in 3D scene panoptic reconstruction on the ScanNet-V2 and ScanNet++ datasets, compared with both NeRF-based and Gaussian-based panoptic reconstruction methods. Moreover, PanopticSplatting can be easily generalized to numerous variants of Gaussian splatting, and we demonstrate its robustness on different Gaussian base models.

Problem

• Open-vocabulary panoptic reconstruction for scene understanding
• End-to-end system to avoid multi-stage errors
• Noisy 2D label handling for consistent 3D segmentation

Innovation

• End-to-end open-vocabulary panoptic Gaussian splatting
• Query-guided Gaussian segmentation with local cross attention
• Label blending and warping for consistent 3D segmentation