Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images

2025-01-08

AI Summary

To address degraded optical remote sensing image scene classification performance under cloud contamination, this paper proposes a transfer learning framework for joint classification of heterogeneous optical and SAR data beneath clouds. Methodologically, we introduce a novel sample-level information regulation mechanism (IRM) to dynamically balance multimodal contributions, and design a knowledge distillation–based collaborative transfer strategy—enabling, for the first time, efficient joint modeling of optical and SAR data in cloudy regions. Key technical components include cloud-simulation augmentation, auxiliary-model-driven modality contribution assessment, sample-level weighted fusion, and dual validation on both real and synthetic cloud-contaminated data. Experiments demonstrate significant performance gains over state-of-the-art methods across multiple cloud-polluted benchmarks; quantitative analysis confirms IRM’s effectiveness in mitigating modality imbalance. The source code and visualization analysis toolkit are publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

In remote sensing scene classification, leveraging the transfer methods with well-trained optical models is an efficient way to overcome label scarcity. However, cloud contamination leads to optical information loss and significant impacts on feature distribution, challenging the reliability and stability of transferred target models. Common solutions include cloud removal for optical data or directly using Synthetic aperture radar (SAR) data in the target domain. However, cloud removal requires substantial auxiliary data for support and pre-training, while directly using SAR disregards the unobstructed portions of optical data. This study presents a scene classification transfer method that synergistically combines multi-modality data, which aims to transfer the source domain model trained on cloudfree optical data to the target domain that includes both cloudy optical and SAR data at low cost. Specifically, the framework incorporates two parts: (1) the collaborative transfer strategy, based on knowledge distillation, enables the efficient prior knowledge transfer across heterogeneous data; (2) the information regulation mechanism (IRM) is proposed to address the modality imbalance issue during transfer. It employs auxiliary models to measure the contribution discrepancy of each modality, and automatically balances the information utilization of modalities during the target model learning process at the sample-level. The transfer experiments were conducted on simulated and real cloud datasets, demonstrating the superior performance of the proposed method compared to other solutions in cloud-covered scenarios. We also verified the importance and limitations of IRM, and further discussed and visualized the modality imbalance problem during the model transfer. Codes are available at https://github.com/wangyuze-csu/ESCCS

Problem

• Cloudy Optical Images
• Scene Classification
• Remote Sensing

Innovation

• Knowledge Distillation
• Adaptation Mechanism
• Cloud-Resilient Optical Imagery