Utilizing Strategic Pre-training to Reduce Overfitting: Baguan -- A Pre-trained Weather Forecasting Model

2025-05-20

AI Summary

To address severe overfitting in AI-based weather forecasting models caused by scarce meteorological data, this paper proposes a strategy-driven pretraining paradigm. Methodologically, it introduces a difficulty-adaptive self-supervised pretraining task explicitly incorporating locality bias to enhance generalization, and constructs Baguan—a unified meteorological foundation model supporting cross-temporal (medium-range to subseasonal) and cross-regional transfer. The model adopts a Siamese autoencoder architecture, integrating spatiotemporal meteorological feature modeling with multi-step fine-tuning. Experiments demonstrate that the approach significantly outperforms conventional numerical weather prediction systems and state-of-the-art AI models on medium-range forecasting tasks. Moreover, pretraining weights exhibit strong transferability to subseasonal-to-seasonal (S2S) and regional forecasting downstream tasks, reducing overfitting rates by over 40%. Key contributions include: (1) a novel locality-biased, difficulty-aware pretraining framework; (2) Baguan, the first unified foundation model for multi-scale meteorological forecasting; and (3) empirical validation of substantial generalization gains under data-scarce conditions.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Weather forecasting has long posed a significant challenge for humanity. While recent AI-based models have surpassed traditional numerical weather prediction (NWP) methods in global forecasting tasks, overfitting remains a critical issue due to the limited availability of real-world weather data spanning only a few decades. Unlike fields like computer vision or natural language processing, where data abundance can mitigate overfitting, weather forecasting demands innovative strategies to address this challenge with existing data. In this paper, we explore pre-training methods for weather forecasting, finding that selecting an appropriately challenging pre-training task introduces locality bias, effectively mitigating overfitting and enhancing performance. We introduce Baguan, a novel data-driven model for medium-range weather forecasting, built on a Siamese Autoencoder pre-trained in a self-supervised manner and fine-tuned for different lead times. Experimental results show that Baguan outperforms traditional methods, delivering more accurate forecasts. Additionally, the pre-trained Baguan demonstrates robust overfitting control and excels in downstream tasks, such as subseasonal-to-seasonal (S2S) modeling and regional forecasting, after fine-tuning.

Problem

• Addressing overfitting in AI-based weather forecasting with limited data
• Developing pre-training strategies to enhance weather model performance
• Introducing Baguan for accurate medium-range and regional forecasting

Innovation

• Strategic pre-training reduces overfitting effectively
• Siamese Autoencoder enables self-supervised learning
• Fine-tuning adapts model for diverse forecasting tasks