Omni-Weather: Unified Multimodal Foundation Model for Weather Generation and Understanding

2025-12-25

AI Summary

Existing weather modeling treats forecasting and mechanistic interpretation as disjoint tasks. This paper introduces the first unified multimodal foundation model for both meteorological generation and understanding. We design a radar encoder coupled with shared multi-head self-attention to jointly optimize generation (e.g., radar echo extrapolation) and understanding tasks (e.g., phenomenon attribution, trend diagnosis). We propose the first meteorological Chain-of-Thought dataset and a causal reasoning training paradigm that explicitly encodes physical causality. A multi-task joint optimization framework enables bidirectional collaborative transfer between generation and understanding. Experiments demonstrate state-of-the-art performance on both task families, with statistically significant positive transfer gains—confirming synergistic learning. To our knowledge, this is the first architecture to simultaneously achieve high-fidelity spatiotemporal forecasting and physically grounded, interpretable causal inference.

Technology Category

• Machine Learning: Multimodal Learning
• Reasoning under Uncertainty: Causality
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-attention mechanism. Moreover, we construct a Chain-of-Thought dataset for causal reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.

Problem

• Unifies weather generation and understanding in a single model
• Integrates radar encoding and shared attention for multimodal tasks
• Enables interpretable outputs through causal reasoning datasets

Innovation

• Unified multimodal model for weather generation and understanding
• Shared self-attention mechanism for unified processing
• Chain-of-Thought dataset for interpretable causal reasoning