GS4: Generalizable Sparse Splatting Semantic SLAM

2025-06-06

AI Summary

Traditional SLAM suffers from low map resolution and poor completeness, while existing Gaussian Splatting (GS)-based SLAM methods exhibit weak generalization and require scene-level optimization. To address these limitations, this paper proposes the first end-to-end, incremental semantic SLAM framework. Methodologically, it employs a shared backbone network to jointly estimate camera poses, sparse Gaussian parameters, and pixel-wise semantics; introduces a single global pose correction step to suppress accumulated drift; and leverages sparse Gaussian splatting for real-time 3D reconstruction and online semantic segmentation. Evaluated on ScanNet, the method achieves state-of-the-art semantic SLAM performance with a 90% reduction in Gaussian count. Furthermore, zero-shot transfer to NYUv2 and TUM RGB-D datasets demonstrates strong cross-scene generalization capability.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: State Estimation
• Machine Learning: Semi-Supervised Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Traditional SLAM algorithms are excellent at camera tracking but might generate lower resolution and incomplete 3D maps. Recently, Gaussian Splatting (GS) approaches have emerged as an option for SLAM with accurate, dense 3D map building. However, existing GS-based SLAM methods rely on per-scene optimization which is time-consuming and does not generalize to diverse scenes well. In this work, we introduce the first generalizable GS-based semantic SLAM algorithm that incrementally builds and updates a 3D scene representation from an RGB-D video stream using a learned generalizable network. Our approach starts from an RGB-D image recognition backbone to predict the Gaussian parameters from every downsampled and backprojected image location. Additionally, we seamlessly integrate 3D semantic segmentation into our GS framework, bridging 3D mapping and recognition through a shared backbone. To correct localization drifting and floaters, we propose to optimize the GS for only 1 iteration following global localization. We demonstrate state-of-the-art semantic SLAM performance on the real-world benchmark ScanNet with an order of magnitude fewer Gaussians compared to other recent GS-based methods, and showcase our model's generalization capability through zero-shot transfer to the NYUv2 and TUM RGB-D datasets.

Problem

• Generalizable Gaussian Splatting for real-time 3D semantic SLAM
• Overcoming per-scene optimization limitations in GS-based SLAM
• Integrating 3D semantic segmentation with efficient Gaussian representation

Innovation

• Generalizable Gaussian Splatting for SLAM
• RGB-D image recognition backbone
• 3D semantic segmentation integration