WeatherGS: 3D Scene Reconstruction in Adverse Weather Conditions via Gaussian Splatting

2024-12-25

AI Summary

Under adverse weather conditions (rain/snow), 3D Gaussian Splatting (3DGS) often misinterprets atmospheric particles and lens water droplets as scene geometry, introducing artifacts and blur. To address this, we propose the first weather-aware Gaussian splatting framework explicitly designed for meteorological interference removal. Our core innovation lies in decoupling weather degradation into two physically grounded components: atmospheric effects (e.g., rain/snow scattering) and lens effects (e.g., water droplet occlusion), modeled respectively by an Atmospheric Effect Filter (AEF) and a Lens Effect Detector (LED). Integrated with mask-guided 3D Gaussian optimization and an enhanced rendering strategy, our method jointly optimizes interference suppression and geometric fidelity. Evaluated on a novel multi-weather benchmark, our approach achieves state-of-the-art performance across PSNR, SSIM, and LPIPS—yielding stable, high-fidelity 3D reconstructions free from weather-induced artifacts.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Hardware-aware ML
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

3D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGS-based framework for reconstructing clear scenes from multi-view images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and raindrops in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods. See project page:https://jumponthemoon.github.io/weather-gs.

Problem

• 3D Reconstruction
• Weather Interference
• 3D Gaussian Splash

Innovation

• WeatherGS
• Adverse Weather Imaging
• 3D Scene Reconstruction