Bridging Artificial Intelligence and Data Assimilation: The Data-driven Ensemble Forecasting System ClimaX-LETKF

2025-12-16

AI Summary

Observational data assimilation remains underexplored in AI-based ensemble weather forecasting. Method: This paper introduces CliaX-LETKF—the first fully data-driven ensemble assimilation system—bypassing numerical models entirely and directly assimilating NCEP global radiosonde and surface observations. It deeply integrates the Local Ensemble Transform Kalman Filter (LETKF) into the deep learning weather model ClimaX and proposes a novel Relaxation-to-Prior Perturbation (RTPP) correction mechanism to expose and mitigate inherent limitations of ML models in recovering atmospheric attractor dynamics. Contribution/Results: CliaX-LETKF achieves multi-year stable operational performance, consistently outperforming the conventional RTPS scheme in both forecast accuracy and ensemble stability. This work establishes a reproducible technical paradigm and provides critical methodological foundations for operationalizing AI-driven ensemble meteorological forecasting.

Technology Category

• Machine Learning: Ensemble Methods
• Intelligent Robots: State Estimation
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While machine learning-based weather prediction (MLWP) has achieved significant advancements, research on assimilating real observations or ensemble forecasts within MLWP models remains limited. We introduce ClimaX-LETKF, the first purely data-driven ML-based ensemble weather forecasting system. It operates stably over multiple years, independently of numerical weather prediction (NWP) models, by assimilating the NCEP ADP Global Upper Air and Surface Weather Observations. The system demonstrates greater stability and accuracy with relaxation to prior perturbation (RTPP) than with relaxation to prior spread (RTPS), while NWP models tend to be more stable with RTPS. RTPP replaces an analysis perturbation with a weighted blend of analysis and background perturbations, whereas RTPS simply rescales the analysis perturbation. Our experiments reveal that MLWP models are less capable of restoring the atmospheric field to its attractor than NWP models. This work provides valuable insights for enhancing MLWP ensemble forecasting systems and represents a substantial step toward their practical applications.

Problem

• Develops a data-driven ensemble weather forecasting system using AI
• Assimilates real observations without relying on numerical weather prediction models
• Compares stability of machine learning and traditional weather forecasting methods

Innovation

• Data-driven ensemble forecasting with ClimaX-LETKF
• Assimilates real observations independently of NWP models
• Uses RTPP for greater stability and accuracy