DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Response

2025-05-27

AI Summary

Existing large vision-language models (VLMs) exhibit weak generalization in complex disaster scenarios, poor cross-sensor adaptability, and low sensitivity to damaged object counting. Method: We introduce DisasterM3—the first multimodal remote sensing vision-language dataset for global disaster response—spanning five continents, ten disaster types, and dual-source (optical + SAR) satellite imagery. It comprises 26,988 temporal image pairs and 123k instruction-response pairs, supporting nine disaster perception and reasoning tasks. We propose a multi-disaster–multi-sensor–multi-task benchmark with a progressive task hierarchy: from object detection and structural damage assessment to long-text disaster report generation. Using instruction tuning and bimodal alignment on models including Qwen-VL and InternVL, we conduct systematic evaluation across 14 state-of-the-art VLMs. Results: After fine-tuning on DisasterM3, four models achieve significant and consistent performance gains across all nine tasks, demonstrating strong cross-sensor and cross-disaster generalization capabilities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large vision-language models (VLMs) have made great achievements in Earth vision. However, complex disaster scenes with diverse disaster types, geographic regions, and satellite sensors have posed new challenges for VLM applications. To fill this gap, we curate a remote sensing vision-language dataset (DisasterM3) for global-scale disaster assessment and response. DisasterM3 includes 26,988 bi-temporal satellite images and 123k instruction pairs across 5 continents, with three characteristics: 1) Multi-hazard: DisasterM3 involves 36 historical disaster events with significant impacts, which are categorized into 10 common natural and man-made disasters. 2)Multi-sensor: Extreme weather during disasters often hinders optical sensor imaging, making it necessary to combine Synthetic Aperture Radar (SAR) imagery for post-disaster scenes. 3) Multi-task: Based on real-world scenarios, DisasterM3 includes 9 disaster-related visual perception and reasoning tasks, harnessing the full potential of VLM's reasoning ability with progressing from disaster-bearing body recognition to structural damage assessment and object relational reasoning, culminating in the generation of long-form disaster reports. We extensively evaluated 14 generic and remote sensing VLMs on our benchmark, revealing that state-of-the-art models struggle with the disaster tasks, largely due to the lack of a disaster-specific corpus, cross-sensor gap, and damage object counting insensitivity. Focusing on these issues, we fine-tune four VLMs using our dataset and achieve stable improvements across all tasks, with robust cross-sensor and cross-disaster generalization capabilities.

Problem

• Addressing VLM challenges in diverse disaster scenes
• Creating a global disaster dataset with multi-sensor data
• Improving VLM performance for damage assessment tasks

Innovation

• Multi-hazard dataset with 36 disaster events
• Combines optical and SAR sensor imagery
• Fine-tunes VLMs for disaster-specific tasks