PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding

2025-03-23

AI Summary

While existing 3D Gaussian Splatting (3DGS)-based methods excel in open-vocabulary scene understanding, they lack instance-level discriminability and produce only text–scene attention heatmaps. This work introduces the first 3DGS-driven panoramic open-vocabulary understanding framework tailored for large-scale indoor scenes, unifying semantic and instance-level 3D segmentation. Our approach addresses geometric inconsistency and language misalignment in superpixel clustering via (1) a language-guided graph-cut algorithm integrated with SAM-enhanced edge affinity modeling, yielding geometry-consistent and language-aligned superprimitives; and (2) a pyramid triplane implicit feature representation fused with multi-view 2D feature clouds for robust 3D decoding. Evaluated on mainstream benchmarks, our method achieves state-of-the-art performance, significantly improving open-vocabulary instance recognition accuracy and cross-scale generalization capability.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently, 3D Gaussian Splatting (3DGS) has shown encouraging performance for open vocabulary scene understanding tasks. However, previous methods cannot distinguish 3D instance-level information, which usually predicts a heatmap between the scene feature and text query. In this paper, we propose PanoGS, a novel and effective 3D panoptic open vocabulary scene understanding approach. Technically, to learn accurate 3D language features that can scale to large indoor scenarios, we adopt the pyramid tri-plane to model the latent continuous parametric feature space and use a 3D feature decoder to regress the multi-view fused 2D feature cloud. Besides, we propose language-guided graph cuts that synergistically leverage reconstructed geometry and learned language cues to group 3D Gaussian primitives into a set of super-primitives. To obtain 3D consistent instance, we perform graph clustering based segmentation with SAM-guided edge affinity computation between different super-primitives. Extensive experiments on widely used datasets show better or more competitive performance on 3D panoptic open vocabulary scene understanding. Project page: href{https://zju3dv.github.io/panogs}{https://zju3dv.github.io/panogs}.

Problem

• Distinguishing 3D instance-level information in open vocabulary scenes
• Modeling latent continuous parametric feature space for large indoor scenarios
• Grouping 3D Gaussian primitives into super-primitives using language-guided graph cuts

Innovation

• Pyramid tri-plane for 3D language feature modeling
• Language-guided graph cuts for super-primitive grouping
• SAM-guided edge affinity for instance segmentation