Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps

2024-06-14

arXiv.org

AI Summary

Existing vision-language models (VLMs) exhibit color perception bias and imprecise spatial localization when interpreting meteorological heatmaps, leading to unreliable explanations for extreme weather event detection (EWED). To address this, we formulate EWED as a vision-language question answering (VQA) task and introduce three key contributions: (1) ClimateIQA—the first domain-specific VQA dataset for meteorology; (2) SPOT, a novel algorithm that enhances precise localization of heatmap color boundaries and critical regions; and (3) Climate-Zoo, a family of meteorology-specialized VLMs. Experiments demonstrate that our approach elevates EWED accuracy from 0% to over 90%, substantially outperforming general-purpose VLMs. All datasets, source code, and pretrained models are publicly released, establishing a reproducible benchmark and foundational infrastructure for AI-driven meteorology.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Real-time detection and prediction of extreme weather protect human lives and infrastructure. Traditional methods rely on numerical threshold setting and manual interpretation of weather heatmaps with Geographic Information Systems (GIS), which can be slow and error-prone. Our research redefines Extreme Weather Events Detection (EWED) by framing it as a Visual Question Answering (VQA) problem, thereby introducing a more precise and automated solution. Leveraging Vision-Language Models (VLM) to simultaneously process visual and textual data, we offer an effective aid to enhance the analysis process of weather heatmaps. Our initial assessment of general-purpose VLMs (e.g., GPT-4-Vision) on EWED revealed poor performance, characterized by low accuracy and frequent hallucinations due to inadequate color differentiation and insufficient meteorological knowledge. To address these challenges, we introduce ClimateIQA, the first meteorological VQA dataset, which includes 8,760 wind gust heatmaps and 254,040 question-answer pairs covering four question types, both generated from the latest climate reanalysis data. We also propose Sparse Position and Outline Tracking (SPOT), an innovative technique that leverages OpenCV and K-Means clustering to capture and depict color contours in heatmaps, providing ClimateIQA with more accurate color spatial location information. Finally, we present Climate-Zoo, the first meteorological VLM collection, which adapts VLMs to meteorological applications using the ClimateIQA dataset. Experiment results demonstrate that models from Climate-Zoo substantially outperform state-of-the-art general VLMs, achieving an accuracy increase from 0% to over 90% in EWED verification. The datasets and models in this study are publicly available for future climate science research: https://github.com/AlexJJJChen/Climate-Zoo.

Problem

• Analyzing meteorological heatmaps with irregular contours and colors
• Improving color identification and spatial localization in VLMs
• Enhancing VLM accuracy for extreme weather interpretation

Innovation

• SPOT algorithm processes irregular colored regions
• ClimateIQA dataset enhances VLM training
• Climate-Zoo suite outperforms existing models