EarthMind: Towards Multi-Granular and Multi-Sensor Earth Observation with Large Multimodal Models

2025-06-02

AI Summary

Current large multimodal models (LMMs) exhibit limited capability in comprehending Earth observation (EO) data, hindering fine-grained environmental dynamics monitoring. To address this, we propose a vision-language framework tailored for multi-granularity, multi-sensor remote sensing understanding. Our method introduces a novel spatial attention prompting mechanism to enhance pixel-level perception, designs an adaptive information-density-weighted cross-modal fusion strategy, and integrates heterogeneous modality alignment, dynamic token reweighting, and multi-scale feature encoding. The framework is jointly trained on human-annotated image-text pairs. We further construct EarthMind-Bench—the first thousand-scale, multi-sensor remote sensing image-text benchmark. Evaluated on EarthMind-Bench, our model surpasses GPT-4o with only 1/10 the parameters; it achieves state-of-the-art performance across multiple public remote sensing benchmarks and, for the first time, enables collaborative optimization of multi-granularity and multi-sensor EO tasks within a unified architecture.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Large Multimodal Models (LMMs) have demonstrated strong performance in various vision-language tasks. However, they often struggle to comprehensively understand Earth Observation (EO) data, which is critical for monitoring the environment and the effects of human activity on it. In this work, we present EarthMind, a novel vision-language framework for multi-granular and multi-sensor EO data understanding. EarthMind features two core components: (1) Spatial Attention Prompting (SAP), which reallocates attention within the LLM to enhance pixel-level understanding; and (2) Cross-modal Fusion, which aligns heterogeneous modalities into a shared space and adaptively reweighs tokens based on their information density for effective fusion. To facilitate multi-sensor fusion evaluation, we propose EarthMind-Bench, a comprehensive benchmark with over 2,000 human-annotated multi-sensor image-question pairs, covering a wide range of perception and reasoning tasks. Extensive experiments demonstrate the effectiveness of EarthMind. It achieves state-of-the-art performance on EarthMind-Bench, surpassing GPT-4o despite being only 4B in scale. Moreover, EarthMind outperforms existing methods on multiple public EO benchmarks, showcasing its potential to handle both multi-granular and multi-sensor challenges in a unified framework.

Problem

• Enhancing Earth Observation data understanding with LMMs
• Addressing multi-granular and multi-sensor fusion challenges
• Improving pixel-level analysis via spatial attention mechanisms

Innovation

• Spatial Attention Prompting enhances pixel-level understanding
• Cross-modal Fusion aligns heterogeneous modalities adaptively
• EarthMind-Bench benchmark for multi-sensor fusion evaluation