DEF: Diffusion-augmented Ensemble Forecasting

2025-06-08

AI Summary

Existing initial-condition perturbation methods are designed for traditional numerical weather prediction (NWP) and poorly adapt to machine learning (ML)-based weather forecasting models, resulting in a lack of generalizable, scalable uncertainty quantification capabilities for ML systems. Method: We propose the first general-purpose initial-perturbation generation framework tailored for ML-based meteorological forecasting. Built upon conditional diffusion models, it iteratively synthesizes physically structured initial-field perturbations with controllable amplitude, incorporating an explicit guidance term to regulate both perturbation magnitude and spectral characteristics. Contribution/Results: The framework is agnostic to neural forecast architectures and achieves significant suppression of long-term error accumulation on the ERA5 dataset. It enhances ensemble spread and improves physical plausibility of forecasts, thereby filling a critical gap in generic stochastic initialization for ML-driven weather prediction.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations

Abstract

We present DEF ( extbf{ul{D}}iffusion-augmented extbf{ul{E}}nsemble extbf{ul{F}}orecasting), a novel approach for generating initial condition perturbations. Modern approaches to initial condition perturbations are primarily designed for numerical weather prediction (NWP) solvers, limiting their applicability in the rapidly growing field of machine learning for weather prediction. Consequently, stochastic models in this domain are often developed on a case-by-case basis. We demonstrate that a simple conditional diffusion model can (1) generate meaningful structured perturbations, (2) be applied iteratively, and (3) utilize a guidance term to intuitivey control the level of perturbation. This method enables the transformation of any deterministic neural forecasting system into a stochastic one. With our stochastic extended systems, we show that the model accumulates less error over long-term forecasts while producing meaningful forecast distributions. We validate our approach on the 5.625$^circ$ ERA5 reanalysis dataset, which comprises atmospheric and surface variables over a discretized global grid, spanning from the 1960s to the present. On this dataset, our method demonstrates improved predictive performance along with reasonable spread estimates.

Problem

• Generating initial condition perturbations for weather prediction
• Transforming deterministic neural forecasts into stochastic systems
• Reducing long-term forecast errors with structured perturbations

Innovation

• Uses conditional diffusion model for perturbations
• Applies perturbations iteratively with guidance
• Transforms deterministic systems into stochastic ones