EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

2024-12-19

arXiv.org

AI Summary

Existing general-purpose and geospatial vision-language models (VLMs) struggle to accommodate the multispectral, multi-temporal, multi-resolution, and high-dimensional heterogeneous characteristics of remote sensing data. To address this, we propose EO-VLMM—the first conversational vision-language model tailored for Earth observation (EO). EO-VLMM introduces a novel architecture supporting multimodal (e.g., RGB, SAR, near-infrared, infrared) and multi-temporal inputs. We construct a large-scale, domain-specific instruction-tuning dataset comprising 11.11 million instruction-response pairs. The model integrates multimodal feature alignment, temporal modeling, and instruction fine-tuning. Evaluated across 44 downstream remote sensing tasks, EO-VLMM consistently outperforms both general-purpose VLMs and state-of-the-art geospatial VLMs, demonstrating substantial improvements in cross-task generalization and fine-grained vision–language reasoning capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Remote Sensing / Geospatial AI
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Automated analysis of vast Earth observation data via interactive Vision-Language Models (VLMs) can unlock new opportunities for environmental monitoring, disaster response, and {resource management}. Existing generic VLMs do not perform well on Remote Sensing data, while the recent Geo-spatial VLMs remain restricted to a fixed resolution and few sensor modalities. In this paper, we introduce EarthDial, a conversational assistant specifically designed for Earth Observation (EO) data, transforming complex, multi-sensory Earth observations into interactive, natural language dialogues. EarthDial supports multi-spectral, multi-temporal, and multi-resolution imagery, enabling a wide range of remote sensing tasks, including classification, detection, captioning, question answering, visual reasoning, and visual grounding. To achieve this, we introduce an extensive instruction tuning dataset comprising over 11.11M instruction pairs covering RGB, Synthetic Aperture Radar (SAR), and multispectral modalities such as Near-Infrared (NIR) and infrared. Furthermore, EarthDial handles bi-temporal and multi-temporal sequence analysis for applications like change detection. Our extensive experimental results on 44 downstream datasets demonstrate that EarthDial outperforms existing generic and domain-specific models, achieving better generalization across various EO tasks. Our source codes and pre-trained models are at https://github.com/hiyamdebary/EarthDial.

Problem

• Enhancing Remote Sensing data analysis via interactive Vision-Language Models
• Overcoming limitations of generic VLMs with multi-sensory, multi-resolution support
• Enabling diverse EO tasks through natural language dialogues

Innovation

• Interactive Vision-Language Models for Earth Observation
• Multi-spectral multi-temporal multi-resolution imagery support
• Extensive instruction tuning dataset with 11.11M pairs