On the Generalizability of Foundation Models for Crop Type Mapping

2024-09-14

arXiv.org

AI Summary

This study systematically evaluates the cross-regional generalization capability of remote sensing foundation models for crop-type mapping, focusing on spatial distribution shifts when transferring from developed to developing countries. Using a harmonized, continent-wide dataset—covering maize, soybean, rice, and wheat across five continents—we conduct in-distribution (ID) and out-of-distribution (OOD) transfer experiments. We first identify the critical advantage of the Sentinel-2–specific self-supervised pre-trained model SSL4EO-S12 in agricultural geographic transfer: it achieves significantly higher OOD overall accuracy than generic ImageNet-pretrained models; with only 100 labeled samples, it attains high overall accuracy, yet ≥900 samples are required to ensure robust performance on long-tailed classes. We propose multispectral normalization and class-balanced fine-tuning strategies, quantifying the impact of annotation scale and label balance on mean accuracy. All data and code are publicly released.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

Foundation models pre-trained using self-supervised learning have shown powerful transfer learning capabilities on various downstream tasks, including language understanding, text generation, and image recognition. The Earth observation (EO) field has produced several foundation models pre-trained directly on multispectral satellite imagery for applications like precision agriculture, wildfire and drought monitoring, and natural disaster response. However, few studies have investigated the ability of these models to generalize to new geographic locations, and potential concerns of geospatial bias -- models trained on data-rich developed nations not transferring well to data-scarce developing nations -- remain. We investigate the ability of popular EO foundation models to transfer to new geographic regions in the agricultural domain, where differences in farming practices and class imbalance make transfer learning particularly challenging. We first select five crop classification datasets across five continents, normalizing for dataset size and harmonizing classes to focus on four major cereal grains: maize, soybean, rice, and wheat. We then compare three popular foundation models, pre-trained on SSL4EO-S12, SatlasPretrain, and ImageNet, using in-distribution (ID) and out-of-distribution (OOD) evaluation. Experiments show that pre-trained weights designed explicitly for Sentinel-2, such as SSL4EO-S12, outperform general pre-trained weights like ImageNet. Furthermore, while only 100 labeled images are sufficient for achieving high overall accuracy, 900 images are required to achieve high average accuracy due to class imbalance. All harmonized datasets and experimental code are open-source and available for download.

Problem

• Evaluate foundation models' geographic generalization for crop mapping.
• Address geospatial bias in transfer learning across diverse regions.
• Compare model performance using specific vs. general pre-trained weights.

Innovation

• Self-supervised learning foundation models
• Transfer learning across geographic regions
• Open-source datasets and experimental code