WildSmoke: Ready-to-Use Dynamic 3D Smoke Assets from a Single Video in the Wild

2025-09-14

AI Summary

This work addresses the challenging problem of reconstructing dynamic 3D smoke assets from uncontrolled monocular outdoor videos and enabling interactive editing. To overcome strong background clutter, absence of multi-view observations, and lack of depth priors in real-world scenes, we propose: (1) background-robust smoke segmentation, (2) physics-constrained particle initialization, and (3) multi-view consistency-aware inference—enabling, for the first time, high-fidelity 4D smoke reconstruction without controlled environments. Our end-to-end pipeline jointly optimizes monocular camera pose estimation, neural rendering-based multi-view synthesis, and differentiable fluid simulation to support semantic-level smoke editing. Quantitatively, our method achieves an average PSNR gain of 2.22 dB over prior approaches on wild videos. We publicly release our model, a curated dataset, and a high-quality 4D smoke asset library to foster further research.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We propose a pipeline to extract and reconstruct dynamic 3D smoke assets from a single in-the-wild video, and further integrate interactive simulation for smoke design and editing. Recent developments in 3D vision have significantly improved reconstructing and rendering fluid dynamics, supporting realistic and temporally consistent view synthesis. However, current fluid reconstructions rely heavily on carefully controlled clean lab environments, whereas real-world videos captured in the wild are largely underexplored. We pinpoint three key challenges of reconstructing smoke in real-world videos and design targeted techniques, including smoke extraction with background removal, initialization of smoke particles and camera poses, and inferring multi-view videos. Our method not only outperforms previous reconstruction and generation methods with high-quality smoke reconstructions (+2.22 average PSNR on wild videos), but also enables diverse and realistic editing of fluid dynamics by simulating our smoke assets. We provide our models, data, and 4D smoke assets at [https://autumnyq.github.io/WildSmoke](https://autumnyq.github.io/WildSmoke).

Problem

• Reconstructing 3D smoke from single wild videos
• Overcoming uncontrolled real-world video challenges
• Enabling interactive smoke simulation and editing

Innovation

• Single video dynamic 3D reconstruction
• Smoke extraction with background removal
• Interactive simulation for smoke editing