SPC-GS: Gaussian Splatting with Semantic-Prompt Consistency for Indoor Open-World Free-view Synthesis from Sparse Inputs

2025-03-16

AI Summary

To address geometric incompleteness and insufficient supervision in 3D Gaussian Splatting (3DGS) for free-viewpoint synthesis in indoor open-world scenes under sparse input conditions, this paper proposes Semantic-Prompt Consistent Gaussian Splatting. The method introduces two key innovations: (1) Scene-layout-guided Gaussian Initialization (SGI), which leverages prior geometric structure to improve the plausibility of initial Gaussian distributions; and (2) SAM2-based cross-view Semantic Prompt Consistency regularization (SPC), jointly enforcing 2D semantic segmentation and 3D rendering constraints to enable semantic-aware reconstruction from sparse views. To our knowledge, this is the first approach to achieve high-fidelity, semantic-guided reconstruction in open-world settings with sparse inputs. Experiments on Replica and ScanNet demonstrate a 3.06 dB PSNR improvement and a 7.3% gain in open-world semantic segmentation mIoU over state-of-the-art sparse-input methods.

Technology Category

• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D Gaussian Splatting-based indoor open-world free-view synthesis approaches have shown significant performance with dense input images. However, they exhibit poor performance when confronted with sparse inputs, primarily due to the sparse distribution of Gaussian points and insufficient view supervision. To relieve these challenges, we propose SPC-GS, leveraging Scene-layout-based Gaussian Initialization (SGI) and Semantic-Prompt Consistency (SPC) Regularization for open-world free view synthesis with sparse inputs. Specifically, SGI provides a dense, scene-layout-based Gaussian distribution by utilizing view-changed images generated from the video generation model and view-constraint Gaussian points densification. Additionally, SPC mitigates limited view supervision by employing semantic-prompt-based consistency constraints developed by SAM2. This approach leverages available semantics from training views, serving as instructive prompts, to optimize visually overlapping regions in novel views with 2D and 3D consistency constraints. Extensive experiments demonstrate the superior performance of SPC-GS across Replica and ScanNet benchmarks. Notably, our SPC-GS achieves a 3.06 dB gain in PSNR for reconstruction quality and a 7.3% improvement in mIoU for open-world semantic segmentation.

Problem

• Improves sparse input performance in 3D Gaussian Splatting.
• Enhances view supervision with semantic-prompt consistency.
• Achieves better reconstruction and semantic segmentation quality.

Innovation

• Scene-layout-based Gaussian Initialization for dense distribution
• Semantic-Prompt Consistency for view supervision enhancement
• 2D and 3D consistency constraints for overlapping regions