WeatherEdit: Controllable Weather Editing with 4D Gaussian Field

2025-05-26

AI Summary

This work addresses the dual challenges of controllability and photorealism in weather simulation for 3D scenes. Methodologically, we propose an end-to-end multi-weather editing framework featuring: (1) a unified diffusion-model adapter jointly trained on multiple weather styles, enabling fine-grained control over weather type and intensity; (2) a TV-Attention mechanism that explicitly models temporal–view interdependencies to ensure spatiotemporal consistency across multi-frame, multi-view sequences; and (3) a physics-informed dynamic 4D Gaussian field that jointly represents and renders the spatiotemporal evolution of diverse atmospheric particles—including rain, snow, and fog. Evaluated on NeRF and 3D Gaussian Splatting (3DGS) reconstructed scenes, our method generates high-fidelity, precisely controllable weather effects across multiple autonomous driving datasets. It significantly enhances the realism and generalization capability of autonomous driving simulators under adverse weather conditions.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

In this work, we present WeatherEdit, a novel weather editing pipeline for generating realistic weather effects with controllable types and severity in 3D scenes. Our approach is structured into two key components: weather background editing and weather particle construction. For weather background editing, we introduce an all-in-one adapter that integrates multiple weather styles into a single pretrained diffusion model, enabling the generation of diverse weather effects in 2D image backgrounds. During inference, we design a Temporal-View (TV-) attention mechanism that follows a specific order to aggregate temporal and spatial information, ensuring consistent editing across multi-frame and multi-view images. To construct the weather particles, we first reconstruct a 3D scene using the edited images and then introduce a dynamic 4D Gaussian field to generate snowflakes, raindrops and fog in the scene. The attributes and dynamics of these particles are precisely controlled through physical-based modelling and simulation, ensuring realistic weather representation and flexible severity adjustments. Finally, we integrate the 4D Gaussian field with the 3D scene to render consistent and highly realistic weather effects. Experiments on multiple driving datasets demonstrate that WeatherEdit can generate diverse weather effects with controllable condition severity, highlighting its potential for autonomous driving simulation in adverse weather. See project page: https://jumponthemoon.github.io/w-edit

Problem

• Controllable weather editing in 3D scenes
• Generating realistic weather effects with adjustable severity
• Ensuring consistent multi-frame and multi-view weather representation

Innovation

• All-in-one adapter integrates multiple weather styles
• Temporal-View attention ensures consistent multi-frame editing
• 4D Gaussian field models dynamic weather particles