Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales

2024-06-19

arXiv.org

AI Summary

To address the challenge of effectively assimilating sparse in-situ observations into full-atmosphere states for kilometer-scale weather forecasting initialization, this paper proposes a score-based generative data assimilation framework. First, an unconditional diffusion model is trained to learn the high-resolution atmospheric prior distribution—using HRRR analyses as ground truth. Then, sparse surface observations (e.g., precipitation, wind fields) are implicitly incorporated into the generative process via score matching. This work presents the first end-to-end, retraining-free generative assimilation method at the kilometer scale; it implicitly encodes multivariate physical constraints without explicit numerical physics modeling, thereby ensuring physical consistency and interpretability of generated fields. Experiments demonstrate that, given only 40 observation sites, the method reduces RMSE for surface variables at withheld sites by 10% relative to the HRRR baseline, while faithfully reproducing realistic meteorological structures—including fronts—with high spatial fidelity.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Data assimilation of observational data into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a central US testbed, we demonstrate the viability of score-based data assimilation in the context of realistically complex km-scale weather. We train an unconditional diffusion model to generate snapshots of a state-of-the-art km-scale analysis product, the High Resolution Rapid Refresh. Then, using score-based data assimilation to incorporate sparse weather station data, the model produces maps of precipitation and surface winds. The generated fields display physically plausible structures, such as gust fronts, and sensitivity tests confirm learnt physics through multivariate relationships. Preliminary skill analysis shows the approach already outperforms a naive baseline of the High-Resolution Rapid Refresh system itself. By incorporating observations from 40 weather stations, 10% lower RMSEs on left-out stations are attained. Despite some lingering imperfections such as insufficiently disperse ensemble DA estimates, we find the results overall an encouraging proof of concept, and the first at km-scale. It is a ripe time to explore extensions that combine increasingly ambitious regional state generators with an increasing set of in situ, ground-based, and satellite remote sensing data streams.

Problem

• Assimilating sparse weather data into km-scale models
• Improving regional weather forecasts using generative methods
• Enhancing precipitation and wind predictions with station observations

Innovation

• Score-based data assimilation for km-scale weather
• Unconditional diffusion model generates HRRR snapshots
• Incorporates sparse station data for precipitation maps