RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting

2025-03-27

AI Summary

Addressing the challenge of physically accurate, real-time controllable rain rendering in open-world scenarios, this paper introduces the first rain synthesis framework integrating physics-based modeling with 3D Gaussian Splatting. Methodologically, we innovatively embed physically grounded raindrop dynamics, shallow water equations, and differentiable rendering into the Gaussian Splatting pipeline, enabling joint modeling of raindrop falling, splashing, surface reflection, and scene-adaptive interactions. Our contributions are threefold: (1) the first framework supporting open-domain deployment, continuous intensity control, and real-time rendering at >30 FPS while preserving physical consistency; (2) significant improvements in physical fidelity and visual realism of rain effects on real-world outdoor and large-scale autonomous driving scenes; and (3) comprehensive outperformance over existing state-of-the-art methods across quantitative and qualitative metrics.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

We consider the problem of adding dynamic rain effects to in-the-wild scenes in a physically-correct manner. Recent advances in scene modeling have made significant progress, with NeRF and 3DGS techniques emerging as powerful tools for reconstructing complex scenes. However, while effective for novel view synthesis, these methods typically struggle with challenging scene editing tasks, such as physics-based rain simulation. In contrast, traditional physics-based simulations can generate realistic rain effects, such as raindrops and splashes, but they often rely on skilled artists to carefully set up high-fidelity scenes. This process lacks flexibility and scalability, limiting its applicability to broader, open-world environments. In this work, we introduce RainyGS, a novel approach that leverages the strengths of both physics-based modeling and 3DGS to generate photorealistic, dynamic rain effects in open-world scenes with physical accuracy. At the core of our method is the integration of physically-based raindrop and shallow water simulation techniques within the fast 3DGS rendering framework, enabling realistic and efficient simulations of raindrop behavior, splashes, and reflections. Our method supports synthesizing rain effects at over 30 fps, offering users flexible control over rain intensity -- from light drizzles to heavy downpours. We demonstrate that RainyGS performs effectively for both real-world outdoor scenes and large-scale driving scenarios, delivering more photorealistic and physically-accurate rain effects compared to state-of-the-art methods. Project page can be found at https://pku-vcl-geometry.github.io/RainyGS/

Problem

• Adding dynamic rain effects to scenes physically-correctly
• Overcoming limitations of NeRF and 3DGS in rain simulation
• Combining physics-based modeling with 3DGS for realistic rain

Innovation

• Integrates physics-based rain simulation with 3DGS
• Enables real-time rain effects at 30 fps
• Supports flexible rain intensity control