RSGaussian:3D Gaussian Splatting with LiDAR for Aerial Remote Sensing Novel View Synthesis

2024-12-24

AI Summary

In aerial remote sensing, 3D Gaussian Splatting (3DGS) suffers from object-scale distortion, floating artifacts, and geospatial misalignment due to insufficient geometric priors. Method: We propose a LiDAR-guided, geometry-aware 3DGS framework. First, we introduce a novel LiDAR point cloud–driven Gaussian growth and splitting mechanism to suppress overgrowth. Second, we design a distortion-aware camera model enabling pixel-level cross-modal registration between LiDAR and imagery. Third, we incorporate depth and planarity consistency losses to enhance multi-view geometric fidelity. Results: Our method achieves a balanced trade-off between visual realism and 3D geometric accuracy. We validate it on AIR-LONGYAN—the first densely annotated, multi-view aerial dataset integrated with high-resolution LiDAR—and demonstrate significant improvements in both reconstruction quality and georegistration precision. The code and dataset are publicly released.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This study presents RSGaussian, an innovative novel view synthesis (NVS) method for aerial remote sensing scenes that incorporate LiDAR point cloud as constraints into the 3D Gaussian Splatting method, which ensures that Gaussians grow and split along geometric benchmarks, addressing the overgrowth and floaters issues occurs. Additionally, the approach introduces coordinate transformations with distortion parameters for camera models to achieve pixel-level alignment between LiDAR point clouds and 2D images, facilitating heterogeneous data fusion and achieving the high-precision geo-alignment required in aerial remote sensing. Depth and plane consistency losses are incorporated into the loss function to guide Gaussians towards real depth and plane representations, significantly improving depth estimation accuracy. Experimental results indicate that our approach has achieved novel view synthesis that balances photo-realistic visual quality and high-precision geometric estimation under aerial remote sensing datasets. Finally, we have also established and open-sourced a dense LiDAR point cloud dataset along with its corresponding aerial multi-view images, AIR-LONGYAN.

Problem

• 3D information deficiency
• object size and position inaccuracy
• geographical alignment error

Innovation

• RSGaussian
• LiDAR-data-integration
• depth-estimation-enhancement