Huge Ensembles Part II: Properties of a Huge Ensemble of Hindcasts Generated with Spherical Fourier Neural Operators

2024-08-02

arXiv.org

AI Summary

This work addresses three key challenges in extreme weather forecasting: insufficient sampling of tail distributions, biased uncertainty quantification, and inadequate representation of internal variability. To this end, we propose HENS—a hyper-scale ensemble forecasting framework comprising 7,424 members. Methodologically, HENS is the first to integrate spherical Fourier neural operators (SFNO), bred-vector initial-condition perturbations, and multi-checkpoint model-parameter perturbations on a ten-million-node spherical grid, enabling a high-fidelity, low-overhead parallel ensemble generation system. Compared with operational systems, HENS achieves comparable physical consistency while reducing computational cost by several orders of magnitude. It significantly improves tail-sampling accuracy for 4σ extreme events, enhances single-member skill and trajectory coverage, and reduces ensemble outlier rates. Moreover, HENS provides a more comprehensive characterization of internal variability, concurrently improving both grid-scale forecast coverage and reliability.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In Part I, we created an ensemble based on Spherical Fourier Neural Operators. As initial condition perturbations, we used bred vectors, and as model perturbations, we used multiple checkpoints trained independently from scratch. Based on diagnostics that assess the ensemble's physical fidelity, our ensemble has comparable performance to operational weather forecasting systems. However, it requires several orders of magnitude fewer computational resources. Here in Part II, we generate a huge ensemble (HENS), with 7,424 members initialized each day of summer 2023. We enumerate the technical requirements for running huge ensembles at this scale. HENS precisely samples the tails of the forecast distribution and presents a detailed sampling of internal variability. For extreme climate statistics, HENS samples events 4$sigma$ away from the ensemble mean. At each grid cell, HENS improves the skill of the most accurate ensemble member and enhances coverage of possible future trajectories. As a weather forecasting model, HENS issues extreme weather forecasts with better uncertainty quantification. It also reduces the probability of outlier events, in which the verification value lies outside the ensemble forecast distribution.

Problem

• Evaluates huge ensemble forecasting performance
• Samples extreme climate statistics accurately
• Improves weather forecast uncertainty quantification

Innovation

• Spherical Fourier Neural Operators
• Bred vectors initialization
• Large-scale ensemble diagnostics