Gaussian-LIC: Real-Time Photo-Realistic SLAM with Gaussian Splatting and LiDAR-Inertial-Camera Fusion

2024-04-10

AI Summary

Existing radiance-field SLAM approaches are primarily limited to small indoor scenes and rely on RGB-D or RGB sensors, exhibiting insufficient robustness in realistic open-world environments—such as those with dynamic objects, varying illumination, or aggressive camera motion. To address these limitations, we propose the first tightly coupled LiDAR-IMU-camera SLAM framework based on 3D Gaussian splatting. Our method introduces a multi-sensor joint initialization scheme that fuses visual triangulation points and LiDAR measurements to initialize 3D Gaussians; incorporates sky modeling and dynamic exposure rendering for enhanced photorealism; and employs nonlinear optimization for online Gaussian parameter refinement, jointly optimizing geometric and radiometric objectives. Implemented efficiently in C++/CUDA, the system achieves real-time performance exceeding 20 FPS. Quantitative evaluation on public benchmarks demonstrates superior pose accuracy over state-of-the-art methods, and remarkably, our self-localized trajectory yields higher-fidelity novel-view synthesis than competing approaches relying on ground-truth poses.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this paper, we present a real-time photo-realistic SLAM method based on marrying Gaussian Splatting with LiDAR-Inertial-Camera SLAM. Most existing radiance-field-based SLAM systems mainly focus on bounded indoor environments, equipped with RGB-D or RGB sensors. However, they are prone to decline when expanding to unbounded scenes or encountering adverse conditions, such as violent motions and changing illumination. In contrast, oriented to general scenarios, our approach additionally tightly fuses LiDAR, IMU, and camera for robust pose estimation and photo-realistic online mapping. To compensate for regions unobserved by the LiDAR, we propose to integrate both the triangulated visual points from images and LiDAR points for initializing 3D Gaussians. In addition, the modeling of the sky and varying camera exposure have been realized for high-quality rendering. Notably, we implement our system purely with C++ and CUDA, and meticulously design a series of strategies to accelerate the online optimization of the Gaussian-based scene representation. Extensive experiments demonstrate that our method outperforms its counterparts while maintaining real-time capability. Impressively, regarding photo-realistic mapping, our method with our estimated poses even surpasses all the compared approaches that utilize privileged ground-truth poses for mapping. Our code will be released on project page https://xingxingzuo.github.io/gaussian_lic.

Problem

• Real-time photo-realistic SLAM for general scenarios
• Robust pose estimation under adverse conditions
• High-quality rendering in unbounded environments

Innovation

• Gaussian Splatting fused with LiDAR-Inertial-Camera
• Triangulated visual and LiDAR points initialize Gaussians
• C++ and CUDA implementation for real-time optimization