MeteorPred: A Meteorological Multimodal Large Model and Dataset for Severe Weather Event Prediction

2025-08-09

AI Summary

Current AI-based weather forecasting systems face three key limitations: scarcity of extreme-event samples, insufficient alignment between meteorological data and textual warning messages, and the inability of multimodal models to effectively capture four-dimensional (4D) spatiotemporal–vertical meteorological dependencies. To address these challenges, we propose the first 4D meteorological multimodal large language model specifically designed for severe weather prediction. We introduce MP-Bench, a large-scale spatiotemporal multimodal benchmark, and design three plug-and-play adaptive fusion modules enabling end-to-end joint modeling of meteorological fields and textual warnings. Our model processes raw 4D meteorological data directly and supports full-parameter fine-tuning and end-to-end training. On MP-Bench, our approach achieves significant improvements in extreme weather detection, localization, and warning generation—advancing the development of fully automated AI-powered weather forecasting systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Timely and accurate severe weather warnings are critical for disaster mitigation. However, current forecasting systems remain heavily reliant on manual expert interpretation, introducing subjectivity and significant operational burdens. With the rapid development of AI technologies, the end-to-end "AI weather station" is gradually emerging as a new trend in predicting severe weather events. Three core challenges impede the development of end-to-end AI severe weather system: (1) scarcity of severe weather event samples; (2) imperfect alignment between high-dimensional meteorological data and textual warnings; (3) existing multimodal language models are unable to handle high-dimensional meteorological data and struggle to fully capture the complex dependencies across temporal sequences, vertical pressure levels, and spatial dimensions. To address these challenges, we introduce MP-Bench, the first large-scale temporal multimodal dataset for severe weather events prediction, comprising 421,363 pairs of raw multi-year meteorological data and corresponding text caption, covering a wide range of severe weather scenarios across China. On top of this dataset, we develop a meteorology multimodal large model (MMLM) that directly ingests 4D meteorological inputs. In addition, it is designed to accommodate the unique characteristics of 4D meteorological data flow, incorporating three plug-and-play adaptive fusion modules that enable dynamic feature extraction and integration across temporal sequences, vertical pressure layers, and spatial dimensions. Extensive experiments on MP-Bench demonstrate that MMLM performs exceptionally well across multiple tasks, highlighting its effectiveness in severe weather understanding and marking a key step toward realizing automated, AI-driven weather forecasting systems. Our source code and dataset will be made publicly available.

Problem

• Scarcity of severe weather event samples for AI training
• Imperfect alignment between high-dimensional data and text warnings
• Existing models fail to handle complex 4D meteorological data

Innovation

• Large-scale multimodal dataset for weather prediction
• 4D meteorology data processing MMLM model
• Adaptive fusion modules for dynamic feature integration