Cloud4D

2025-11-24

AI Summary

Current kilometer-scale numerical weather models struggle to accurately resolve cloud microphysical structures and extreme weather elements—such as intense precipitation, gusty winds, and surface irradiance—primarily due to the lack of high spatiotemporal-resolution in situ observational constraints. To address this, we propose the first learning-based four-dimensional (4D) cloud-state reconstruction framework. It synergistically integrates multi-view, ground-based, synchronized camera imagery; a source-consistency-guided 2D-to-3D Transformer network; and a temporal wind-field tracking algorithm, enabling physically consistent joint retrieval of three-dimensional liquid water content distribution and horizontal wind fields. Evaluated over a two-month field campaign with six synchronized cameras, the framework achieves 25-m spatial resolution and 5-s temporal resolution—improving upon satellite-based observations by an order of magnitude—and yields wind-field retrieval errors below 10% relative to radar reference data. This significantly enhances fine-scale observational capability for extreme weather phenomena.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Image and Video Retrieval
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four-dimensional cloud state using only synchronized ground-based cameras. Leveraging a homography-guided 2D-to-3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10%$) against collocated radar measurements. Code and data are available on our project page https://cloud4d.jacob-lin.com/.

Problem

• Reconstructs 4D cloud states using ground cameras
• Infers 3D liquid water content at high resolution
• Estimates horizontal wind vectors from temporal tracking

Innovation

• Uses synchronized ground-based cameras for 4D reconstruction
• Employs homography-guided 2D-to-3D transformer for inference
• Achieves high-resolution 3D liquid water content tracking