CLIPPan: Adapting CLIP as A Supervisor for Unsupervised Pansharpening

2025-11-14

AI Summary

Supervised pansharpening methods suffer from poor generalization due to domain shift between synthetically degraded training data and real-world multispectral–panchromatic image pairs. To address this, we propose CLIPPan—the first unsupervised pansharpening framework leveraging vision-language pretraining via CLIP. CLIPPan introduces lightweight fine-tuning to adapt CLIP to multispectral and panchromatic modalities, and designs a semantic-aware loss that aligns the fusion process with text prompts (e.g., “Wald/Khan protocol description”), enabling label-free, semantics-guided reconstruction. Evaluated across multiple real-world remote sensing datasets and diverse backbone architectures, CLIPPan consistently achieves superior spatial detail preservation and spectral fidelity compared to state-of-the-art supervised and unsupervised methods. It establishes a new benchmark for unsupervised full-resolution pansharpening, demonstrating robust cross-dataset generalization without ground-truth references.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite remarkable advancements in supervised pansharpening neural networks, these methods face domain adaptation challenges of resolution due to the intrinsic disparity between simulated reduced-resolution training data and real-world full-resolution scenarios.To bridge this gap, we propose an unsupervised pansharpening framework, CLIPPan, that enables model training at full resolution directly by taking CLIP, a visual-language model, as a supervisor. However, directly applying CLIP to supervise pansharpening remains challenging due to its inherent bias toward natural images and limited understanding of pansharpening tasks. Therefore, we first introduce a lightweight fine-tuning pipeline that adapts CLIP to recognize low-resolution multispectral, panchromatic, and high-resolution multispectral images, as well as to understand the pansharpening process. Then, building on the adapted CLIP, we formulate a novel extit{loss integrating semantic language constraints}, which aligns image-level fusion transitions with protocol-aligned textual prompts (e.g., Wald's or Khan's descriptions), thus enabling CLIPPan to use language as a powerful supervisory signal and guide fusion learning without ground truth. Extensive experiments demonstrate that CLIPPan consistently improves spectral and spatial fidelity across various pansharpening backbones on real-world datasets, setting a new state of the art for unsupervised full-resolution pansharpening.

Problem

• Bridging domain adaptation gap between simulated and real-world pansharpening scenarios
• Adapting CLIP model to understand multispectral imagery and pansharpening process
• Developing unsupervised full-resolution training method using language as supervision

Innovation

• Adapts CLIP for unsupervised pansharpening supervision
• Fine-tunes CLIP to understand multispectral and panchromatic images
• Uses language prompts as loss constraints without ground truth