How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

2026-06-11

AI Summary

This study addresses the lack of systematic understanding regarding the transfer behavior of existing self-supervised remote sensing vision models on downstream tasks. We systematically evaluate six representative geospatial foundation models across classification, regression, and segmentation tasks, employing layer probing, Centered Kernel Alignment (CKA) analysis, and diverse fine-tuning strategies. Our findings reveal that task-relevant information is predominantly concentrated in the intermediate layers of Vision Transformers (ViTs), while fine-tuning primarily affects the first layer of the MLP blocks. Furthermore, we elucidate the root causes behind the performance variability of GeoFM across different benchmarks, demonstrating that decoder architecture and adaptation strategies exert an influence on segmentation performance comparable to that of model selection itself. This work establishes a new paradigm for evaluating and adapting foundation models in remote sensing.

Abstract

Self-supervised geospatial foundation models (GeoFMs) learn transferable representations from remote sensing data, but their downstream behavior is difficult to characterize. We study six representative GeoFMs spanning joint-embedding, reconstruction, and multimodal pretraining families, and evaluate transfer across classification, regression, and segmentation benchmarks under different label availability and downstream pipelines. We find that model rankings change across tasks and adaptation settings. Layerwise probing shows that, in most cases, task-relevant information is more accessible in intermediate transformer blocks compared to final-layer embeddings, and that GeoFMs exhibit distinct depthwise profiles. In segmentation case studies on PASTIS and Sen1Floods11, downstream adaptation settings such as decoder design and fine-tuning can be as impactful as the choice of GeoFM, and standard dense-prediction heads may be poorly aligned with how GeoFMs organize information over depth. Finally, CKA analysis on case studies shows that fine-tuning does not rewrite GeoFMs uniformly across depth, and the strongest changes are localized to the first linear layer of the MLP in ViT blocks. These results help explain why GeoFM rankings shift across benchmarks and motivate more representation-aware evaluation and adaptation strategies.

Problem

• self-supervised learning
• remote sensing
• foundation models
• transfer learning
• representation analysis

Innovation

• self-supervised learning
• geospatial foundation models
• transferability
• layerwise probing
• representation analysis