AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities

2024-12-18

AI Summary

Existing geospatial models are constrained by fixed input configurations, limiting their adaptability to the high heterogeneity of remote sensing data in resolution, scale, and modality. To address this, we propose AnySat—a unified Earth observation model featuring a novel Joint Embedding Predictive Architecture (JEPA) and a scale-adaptive spatial encoder, enabling self-supervised joint learning across multi-resolution, multi-scale, and multi-modal remote sensing data. We introduce GeoPlex, the first large-scale, multi-source, multi-task geospatial benchmark, and adopt a two-stage training paradigm: cross-dataset joint pretraining followed by task-specific probe fine-tuning to enhance generalization. Evaluated on all GeoPlex test sets and six external benchmarks, AnySat achieves state-of-the-art performance across diverse downstream tasks—including land cover mapping, tree species identification, crop classification, change detection, climate classification, and segmentation of flood, burn, and deforestation areas—demonstrating unprecedented versatility and practical utility in Earth observation.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Geospatial models must adapt to the diversity of Earth observation data in terms of resolutions, scales, and modalities. However, existing approaches expect fixed input configurations, which limits their practical applicability. We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and scale-adaptive spatial encoders, allowing us to train a single model on highly heterogeneous data in a self-supervised manner. To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of $5$ multimodal datasets with varying characteristics and $11$ distinct sensors. We then train a single powerful model on these diverse datasets simultaneously. Once fine-tuned or probed, we reach state-of-the-art results on the test sets of GeoPlex and for $6$ external datasets across various environment monitoring tasks: land cover mapping, tree species identification, crop type classification, change detection, climate type classification, and segmentation of flood, burn scar, and deforestation. The code and models are available at https://github.com/gastruc/AnySat.

Problem

• Adapting geospatial models to diverse Earth observation data resolutions and scales
• Overcoming limitations of fixed input configurations in existing approaches
• Achieving state-of-the-art results across multiple environment monitoring tasks

Innovation

• Multimodal model with JEPA architecture
• Scale-adaptive spatial encoders
• Self-supervised heterogeneous data training