Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement

2026-07-02

AI Summary

Existing 2D instance segmentation models often produce fragmented and inconsistent masks across multiple views, hindering reliable 3D scene understanding. This work proposes a multi-cue guided approach for generating cross-view consistent 2D instance masks by integrating semantic, geometric, and structural information, along with an identity matching mechanism to align instances across viewpoints. These consistent masks are then leveraged to guide the optimization of a 3D Gaussian Splatting feature field, marking the first effective incorporation of instance-level consistency into the Gaussian Splatting framework. Experiments demonstrate that the proposed method significantly improves cross-view mask consistency and 3D instance segmentation stability while preserving high-quality photometric reconstruction, thereby enabling robust downstream editing tasks.

Abstract

Reliable instance-level scene understanding is a fundamental prerequisite for object-level interactions and high-fidelity 3D representations. While current methods often leverage 2D foundation segmentation models to obtain these priors, their 2D-centric design typically yields fragmented masks and inconsistent predictions across different views. To address these issues, we propose a novel framework that produces consistent 2D instance masks to guide the optimization of 3D Gaussian Splatting (3DGS) feature fields. Our framework consists of three main stages. (1) Multi-Cue Extraction that generates synergistic semantic, geometric, and structural priors from input images. (2) Multi-Cue-Guided Mask Merging process that consolidates fragmented masks using a composite merge score derived from semantic, depth, and edge cues. (3) Cross-View Mask Matching that establishes globally consistent identity assignments across all viewpoints. By transforming viewpoint-specific segments into coherent 3D primitives, our approach enables stable 3D instance segmentation and effective downstream editing tasks. Experiments demonstrate that our method significantly improves cross-view consistency and segmentation stability over existing baselines while maintaining high-fidelity photometric reconstruction.

Problem

• 3D Gaussian Splatting
• instance segmentation
• cross-view consistency
• mask refinement
• scene understanding

Innovation

• 3D Gaussian Splatting
• instance segmentation
• multi-cue fusion
• cross-view consistency
• mask refinement