How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?

2025-01-21

AI Summary

This study investigates how the geographic distribution of pretraining data affects the performance of geospatial foundation models (GFMs). To this end, we systematically sample pretraining subsets from multisource remote sensing imagery with varying spatial compositions—specifically, globally balanced versus regionally specialized distributions—and evaluate their generalization across downstream tasks—including climate modeling, agricultural monitoring, and disaster response—under two distinct GFM architectures. Our analysis reveals, for the first time, that geographic diversity and global coverage are critical determinants of GFM robustness. While globally balanced sampling generally yields superior generalization, the optimal strategy is architecture-dependent. These findings establish reproducible, data-distribution-aware guidelines for constructing high-generalization GFMs, directly informing best practices in geospatial representation learning.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Remote Sensing / Geospatial AI
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

Foundation models have made rapid advances in many domains including Earth observation, where Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. However, recent works from other domains show that the pre-training data distribution is an important factor influencing the performance of the foundation models. With this motivation, our research explores how the geographic distribution of pre-training data affects the performance of GFMs. We evaluated several pre-training data distributions by sampling different compositions from a global data pool. Our experiments with two GFMs on downstream tasks indicate that balanced and globally representative data compositions often outperform region-specific sampling, highlighting the importance of diversity and global coverage in pre-training data. Our results suggest that the most appropriate data sampling technique may depend on the specific GFM architecture. These findings will support the development of robust GFMs by incorporating quality pre-training data distributions, ultimately improving machine learning solutions for Earth observation.

Problem

• Pretrained data distribution
• Geospatial applications
• Earth observation

Innovation

• Geographical Foundational Models
• Diverse Training Data
• Balanced Sampling Strategy