LightWeather: Harnessing Absolute Positional Encoding to Efficient and Scalable Global Weather Forecasting

2024-08-19

arXiv.org

AI Summary

To address the high parameter count, slow training, and poor scalability of Transformers in atmospheric time-series forecasting (ATSF), this paper proposes a minimalist yet highly efficient paradigm. We theoretically prove and empirically validate that an absolute positional encoding—integrating latitude, longitude, and UTC time—can fully capture spatiotemporal weather dynamics, effectively replacing self-attention. The resulting model consists solely of a multilayer perceptron (MLP) augmented with a custom geospatial-temporal positional encoding, containing fewer than 30K parameters and requiring less than one hour of training on a single GPU. Evaluated on a large-scale global meteorological benchmark, our method achieves state-of-the-art (SOTA) accuracy while incurring significantly lower computational overhead than existing deep learning approaches. This work establishes a novel paradigm for high-accuracy, low-cost, and scalable global weather forecasting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recently, Transformers have gained traction in weather forecasting for their capability to capture long-term spatial-temporal correlations. However, their complex architectures result in large parameter counts and extended training times, limiting their practical application and scalability to global-scale forecasting. This paper aims to explore the key factor for accurate weather forecasting and design more efficient solutions. Interestingly, our empirical findings reveal that absolute positional encoding is what really works in Transformer-based weather forecasting models, which can explicitly model the spatial-temporal correlations even without attention mechanisms. We theoretically prove that its effectiveness stems from the integration of geographical coordinates and real-world time features, which are intrinsically related to the dynamics of weather. Based on this, we propose LightWeather, a lightweight and effective model for station-based global weather forecasting. We employ absolute positional encoding and a simple MLP in place of other components of Transformer. With under 30k parameters and less than one hour of training time, LightWeather achieves state-of-the-art performance on global weather datasets compared to other advanced DL methods. The results underscore the superiority of integrating spatial-temporal knowledge over complex architectures, providing novel insights for DL in weather forecasting.

Problem

• Reducing excessive parameters and training time in atmospheric forecasting models
• Replacing complex Transformer architectures with simpler spatial-temporal integration
• Achieving accurate forecasting using lightweight MLP with position embedding

Innovation

• Replaces Transformers with spatial-temporal position embedding
• Uses only STPE and MLP for lightweight architecture
• Integrates geographical coordinates and temporal features