SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understanding

2026-03-02

AI Summary

Existing methods for 3D semantic reconstruction often rely on dense multi-view inputs and scene-specific optimization, limiting their efficiency and generalization in real-world scenarios. This work proposes SemGS, a feed-forward framework that directly reconstructs semantic 3D Gaussian representations from sparse input views and enables semantic map synthesis from novel viewpoints. SemGS employs a dual-branch architecture with shared shallow CNNs, a camera-aware attention mechanism, and a dual-Gaussian representation, complemented by a region-smoothness loss to enhance semantic consistency—all without requiring test-time optimization. The method achieves state-of-the-art performance across multiple benchmarks, demonstrating both fast inference speed and strong generalization capabilities in both synthetic and real-world scenes.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Semantic understanding of 3D scenes is essential for robots to operate effectively and safely in complex environments. Existing methods for semantic scene reconstruction and semantic-aware novel view synthesis often rely on dense multi-view inputs and require scene-specific optimization, limiting their practicality and scalability in real-world applications. To address these challenges, we propose SemGS, a feed-forward framework for reconstructing generalizable semantic fields from sparse image inputs. SemGS uses a dual-branch architecture to extract color and semantic features, where the two branches share shallow CNN layers, allowing semantic reasoning to leverage textural and structural cues in color appearance. We also incorporate a camera-aware attention mechanism into the feature extractor to explicitly model geometric relationships between camera viewpoints. The extracted features are decoded into dual-Gaussians that share geometric consistency while preserving branch-specific attributes, and further rasterized to synthesize semantic maps under novel viewpoints. Additionally, we introduce a regional smoothness loss to enhance semantic coherence. Experiments show that SemGS achieves state-of-the-art performance on benchmark datasets, while providing rapid inference and strong generalization capabilities across diverse synthetic and real-world scenarios.

Problem

• semantic scene reconstruction
• novel view synthesis
• sparse views
• generalizable scene understanding
• 3D Gaussian splatting

Innovation

• Semantic 3D Gaussian Splatting
• Sparse-view Reconstruction
• Feed-forward Framework
• Camera-aware Attention
• Dual-branch Architecture