Towards Fair Comparisons of AI- and Physics-Based Weather Models for Extreme Events via the Weighted Potential CRPS

2026-06-19

AI Summary

This study addresses the challenge of fairly evaluating artificial intelligence (AI) models against physics-based numerical weather prediction (NWP) systems in forecasting extreme weather events. To this end, the authors propose a weighted version of the continuous ranked probability score in latent space (Weighted PCRPS), coupled with Isotonic Distributional Regression (IDR) to convert deterministic forecasts into probabilistic predictions for rigorous assessment. The weighting scheme emphasizes extreme events, while IDR ensures evaluation fairness through its optimality properties. Comprehensive comparisons on the WeatherBench 2 dataset among leading AI models—including GraphCast, Pangu-Weather, and FuXi—and ECMWF’s high-resolution NWP system demonstrate that FuXi achieves overall superior performance across key extreme meteorological variables such as pressure, temperature, wind speed, and precipitation, highlighting the potential of AI-based approaches to surpass traditional NWP in extreme weather forecasting.

Abstract

We study whether deterministic AI weather prediction (AIWP) models issue more informative forecasts for extreme weather events than deterministic numerical weather prediction (NWP) models. The deterministic model output is subjected to statistical post-processing via isotonic distributional regression (IDR), or EasyUQ, before the resulting probabilistic forecasts are assessed using weighted versions of the continuous ranked probability score (CRPS). This extends the Potential CRPS (PCRPS) measure proposed by Gneiting et al. (2026) to focus on extreme outcomes. Since IDR exhibits optimality properties with respect to weighted versions of the CRPS, the proposed approach inherits desirable properties of the PCRPS, and, in particular, facilitates fair comparisons between data-driven and physics-based models when forecasting extreme weather events. We apply this evaluation framework to forecasts in the WeatherBench 2 dataset issued by the AIWP models GraphCast, Pangu-Weather, and FuXi, with the ECMWF's high-resolution NWP model serving as a physics-based reference. The forecast models are compared when predicting mean sea level pressure, temperature, wind speed, and precipitation extremes, defined as exceedances or non-exceedances of thresholds obtained from historical observation data. We additionally study forecast performance when predicting record-breaking events, though the ordering of the different methods is largely insensitive to the thresholds on which emphasis is placed. We find that AIWP models, particularly FuXi, result in the most informative forecasts for extreme weather events across most settings, suggesting that AIWP models have the potential to outperform NWP models when forecasting extremes.

Problem

• AI weather prediction
• numerical weather prediction
• extreme weather events
• forecast evaluation
• fair comparison

Innovation

• Weighted Potential CRPS
• Isotonic Distributional Regression
• Extreme Weather Forecasting
• AI Weather Prediction
• Fair Model Comparison