A Multimodal, Multilingual, and Multidimensional Pipeline for Fine-grained Crowdsourcing Earthquake Damage Evaluation

2025-06-03

AI Summary

Social media data—multimodal and unstructured—pose significant challenges for fine-grained, real-time earthquake damage assessment. Method: We propose the first 3M framework (Multilingual, Multi-distance, Multi-modal) to systematically characterize multimodal large language model (MLLM) performance across language, epicentral distance, and modality combinations. We introduce EarthQuake-Social-Bench, the first open-source benchmark for social-media-driven earthquake impact assessment, alongside an integrated codebase. Our MLLM analysis pipeline jointly models visual content, cross-lingual text semantics, and geospatial context to enable multimodal, multi-scale reasoning. Contribution/Results: Evaluated on two real-world earthquakes, our approach achieves strong correlation (r > 0.82) between model outputs and ground-truth damage metrics—substantially outperforming unimodal baselines—and enables minute-level generation of high-resolution damage heatmaps.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Rapid, fine-grained disaster damage assessment is essential for effective emergency response, yet remains challenging due to limited ground sensors and delays in official reporting. Social media provides a rich, real-time source of human-centric observations, but its multimodal and unstructured nature presents challenges for traditional analytical methods. In this study, we propose a structured Multimodal, Multilingual, and Multidimensional (3M) pipeline that leverages multimodal large language models (MLLMs) to assess disaster impacts. We evaluate three foundation models across two major earthquake events using both macro- and micro-level analyses. Results show that MLLMs effectively integrate image-text signals and demonstrate a strong correlation with ground-truth seismic data. However, performance varies with language, epicentral distance, and input modality. This work highlights the potential of MLLMs for disaster assessment and provides a foundation for future research in applying MLLMs to real-time crisis contexts. The code and data are released at: https://github.com/missa7481/EMNLP25_earthquake

Problem

• Rapid fine-grained earthquake damage assessment lacks ground sensors
• Social media data is unstructured and hard to analyze traditionally
• Multimodal models need evaluation for language and distance variations

Innovation

• Multimodal pipeline for earthquake damage evaluation
• Uses MLLMs to integrate image-text signals
• Evaluates models across languages and modalities