Controllable Weather Synthesis and Removal with Video Diffusion Models

2025-05-01

AI Summary

Synthesizing and removing realistic, controllable weather effects (e.g., rain, snow, fog, clouds) in in-the-wild videos remains challenging: physics-based simulation lacks scalability, while generative approaches suffer from limited photorealism and fine-grained controllability. Method: We propose WeatherWeaver—the first controllable diffusion model for video weather editing. It introduces a novel multi-source heterogeneous data distillation strategy, jointly leveraging synthetic videos, generative image editing outputs, and self-supervised annotations on real-world videos to alleviate the scarcity of paired training data. A tunable conditional modeling mechanism enables single- or multi-weather editing without requiring 3D priors. Results: Extensive experiments demonstrate state-of-the-art performance on both weather synthesis and removal tasks, achieving superior photorealism, physical plausibility, and cross-scene generalization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generating realistic and controllable weather effects in videos is valuable for many applications. Physics-based weather simulation requires precise reconstructions that are hard to scale to in-the-wild videos, while current video editing often lacks realism and control. In this work, we introduce WeatherWeaver, a video diffusion model that synthesizes diverse weather effects -- including rain, snow, fog, and clouds -- directly into any input video without the need for 3D modeling. Our model provides precise control over weather effect intensity and supports blending various weather types, ensuring both realism and adaptability. To overcome the scarcity of paired training data, we propose a novel data strategy combining synthetic videos, generative image editing, and auto-labeled real-world videos. Extensive evaluations show that our method outperforms state-of-the-art methods in weather simulation and removal, providing high-quality, physically plausible, and scene-identity-preserving results over various real-world videos.

Problem

• Generating controllable weather effects in videos
• Overcoming lack of realism in current video editing
• Addressing scarcity of paired training data

Innovation

• Video diffusion model for weather synthesis
• Control over weather intensity and blending
• Combined synthetic and real-world training data