Local Off-Grid Weather Forecasting with Multi-Modal Earth Observation Data

2024-10-16

AI Summary

Large-scale gridded weather forecasts often fail to resolve near-surface local meteorological conditions, limiting their utility in high-temporal-demand applications such as wildfire management and renewable energy dispatch. To address this, we propose an end-to-end, site-specific fine-grained forecasting framework. Our approach introduces a novel station-level multimodal tokenization scheme that jointly encodes gridded forecasts and localized in-situ observations. We further design a spatiotemporal self-attention mechanism enabling dynamic cross-station information aggregation and—critically—demonstrate for the first time that historical observations can induce phase transitions in forecast evolution, substantially improving nowcasting through medium-range prediction accuracy. Evaluated across multiple stations in the U.S. Northeast, our method consistently outperforms leading data-driven and physics-based models, reducing key error metrics by up to 80%. This advances reliable decision support for high-risk operational scenarios.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Urgent applications like wildfire management and renewable energy generation require precise, localized weather forecasts near the Earth's surface. However, forecasts produced by machine learning models or numerical weather prediction systems are typically generated on large-scale regular grids, where direct downscaling fails to capture fine-grained, near-surface weather patterns. In this work, we propose a multi-modal transformer model trained end-to-end to downscale gridded forecasts to off-grid locations of interest. Our model directly combines local historical weather observations (e.g., wind, temperature, dewpoint) with gridded forecasts to produce locally accurate predictions at various lead times. Multiple data modalities are collected and concatenated at station-level locations, treated as a token at each station. Using self-attention, the token corresponding to the target location aggregates information from its neighboring tokens. Experiments using weather stations across the Northeastern United States show that our model outperforms a range of data-driven and non-data-driven off-grid forecasting methods. They also reveal that direct input of station data provides a phase shift in local weather forecasting accuracy, reducing the prediction error by up to 80% compared to pure gridded data based models. This approach demonstrates how to bridge the gap between large-scale weather models and locally accurate forecasts to support high-stakes, location-sensitive decision-making.

Problem

• Downscaling gridded forecasts to off-grid locations
• Combining local observations with gridded forecasts
• Improving near-surface weather prediction accuracy

Innovation

• Multi-modal transformer model for local forecasting
• Combines local observations with gridded forecasts
• Self-attention aggregates neighboring station data