SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAM

2024-03-12

arXiv.org

AI Summary

This work addresses key challenges in dense semantic SLAM—low 3D semantic mapping accuracy, camera tracking drift, and poor novel-view rendering quality—by proposing the first end-to-end framework integrating semantic feature embedding into 3D Gaussian representations. Methodologically: (1) we introduce a semantic-enhanced 3D Gaussian representation, encoding pixel-level semantic features into Gaussian ellipsoid attributes; (2) we design a feature-level supervised loss to jointly optimize geometric and semantic consistency; and (3) we propose semantic-correlation-driven multi-frame bundle adjustment to significantly suppress pose accumulation errors. Evaluated on Replica and ScanNet, our method outperforms state-of-the-art NeRF-based SLAM approaches across mapping accuracy, tracking robustness, and novel-view synthesis quality, while enabling high-fidelity dense semantic segmentation and editable semantic 3D map reconstruction.

Technology Category

• Computer Vision: Representation Learning for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We propose SemGauss-SLAM, a dense semantic SLAM system utilizing 3D Gaussian representation, that enables accurate 3D semantic mapping, robust camera tracking, and high-quality rendering simultaneously. In this system, we incorporate semantic feature embedding into 3D Gaussian representation, which effectively encodes semantic information within the spatial layout of the environment for precise semantic scene representation. Furthermore, we propose feature-level loss for updating 3D Gaussian representation, enabling higher-level guidance for 3D Gaussian optimization. In addition, to reduce cumulative drift in tracking and improve semantic reconstruction accuracy, we introduce semantic-informed bundle adjustment leveraging multi-frame semantic associations for joint optimization of 3D Gaussian representation and camera poses, leading to low-drift tracking and accurate mapping. Our SemGauss-SLAM method demonstrates superior performance over existing radiance field-based SLAM methods in terms of mapping and tracking accuracy on Replica and ScanNet datasets, while also showing excellent capabilities in high-precision semantic segmentation and dense semantic mapping.

Problem

• Enables accurate 3D semantic mapping and camera tracking
• Incorporates semantic features into 3D Gaussian representation
• Reduces tracking drift with semantic-informed bundle adjustment

Innovation

• 3D Gaussian representation with semantic embedding
• Feature-level loss for Gaussian optimization
• Semantic-informed bundle adjustment for tracking