🤖 AI Summary
Remote sensing image fusion models suffer from limited generalizability due to scarce authentic ground-truth data and domain shifts across heterogeneous sensors. To address this, we introduce, for the first time, a foundation model paradigm into remote sensing fusion, proposing a large-scale pretraining framework grounded in spatial-spectral priors. Our method synthesizes a highly diverse dataset by applying realistic degradations—including blur, noise, and downsampling—to ImageNet and SkyScript images. This enables effective pretraining across multiple architectures, including CNNs, Transformers, and Mamba. The resulting model achieves zero-shot and few-shot pan-sharpening, outperforming state-of-the-art methods on six major satellite datasets (e.g., WorldView). Remarkably, it adapts to unseen sensor domains with fine-tuning on merely a single real-world image. Our work establishes a new benchmark for cross-domain generalization in remote sensing image fusion.
📝 Abstract
Existing deep learning methods for remote sensing image fusion often suffer from poor generalization when applied to unseen datasets due to the limited availability of real training data and the domain gap between different satellite sensors. To address this challenge, we explore the potential of foundation models by proposing a novel pretraining strategy that leverages large-scale simulated datasets to learn robust spatial-spectral priors. Specifically, our approach first constructs diverse simulated datasets by applying various degradation operations (blur, noise, downsampling) and augmentations (bands generation, channel shuffling, high-pass filtering, color jittering, etc.) to natural images from ImageNet and remote sensing images from SkyScript. We then pretrain fusion models on these simulated data to learn generalizable spatial-spectral representations. The pretrained models are subsequently evaluated on six datasets (WorldView-2/3/4, IKONOS, QuickBird, GaoFen-2) using zero-shot and one-shot paradigms, with both full- and freeze-tuning approaches for fine-tuning. Extensive experiments on different network architectures including convolutional neural networks, Transformer, and Mamba demonstrate that our pretraining strategy significantly improves generalization performance across different satellite sensors and imaging conditions for various fusion models. The pretrained models achieve superior results in zero-shot scenarios and show remarkable adaptation capability with minimal real data in one-shot settings. Our work provides a practical solution for cross-domain pansharpening, establishes a new benchmark for generalization in remote sensing image fusion tasks, and paves the way for leveraging foundation models through advanced training strategies.