Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction

2026-06-25

AI Summary

This work addresses the challenges of multimodal image fusion under adverse weather conditions, where severe image degradation, feature distortion, and difficulties in modeling cross-modal complementarity hinder performance. To tackle these issues, the authors propose a mask-guided fusion framework that simplifies training through pseudo-ground-truth supervision and introduces a dynamic mask generation mechanism based on the mapping relationship between fused outputs and source images. This mechanism enables the network to selectively focus on informative features during cross-modal interaction. Furthermore, by integrating mask-guided learning with a task-coupled, degradation-aware strategy, the method jointly optimizes feature restoration and fusion. Notably, it incorporates a novel mask-guided cross-modal cross-attention module, achieving consistent and significant improvements over state-of-the-art approaches on both synthetic and real-world datasets in terms of visual quality, quantitative metrics, and downstream task performance.

Abstract

Multi-modality image fusion (MMIF) enhances scene representation by exploiting complementary cues from different modalities. Adverse weather, however, causes significant image degradation, disrupting feature representation and requiring simultaneous feature restoration and cross-modal complementarity. Existing methods often struggle with effective representation learning under such conditions, limiting their practical performance. To address these challenges, we propose a mask-guided MMIF method that integrates feature restoration and interaction. We first introduce "Pseudo Ground Truth" to simplify training, promoting faster and more effective feature learning. Then, we design a mask generation mechanism based on the mapping relationship between the fused result and the source images, quantifying the relative contribution of each modality during the fusion process. By incorporating the proposed mask-guided cross-modal cross-attention mechanism, the network is encouraged to selectively attend to informative features during modality interaction, mitigating the risk of overfitting to the static distribution of the "Pseudo Ground Truth". Additionally, we propose a mask-guided learning strategy and a task-coupled degradation-aware learning strategy to balance feature restoration and interaction. Extensive experiments on synthetic and real-world datasets demonstrate that our method surpasses state-of-the-art approaches in visual quality, quantitative metrics, and downstream tasks. The source code is available at https://github.com/ixilai/AMG-Fuse.

Problem

• Multi-modality Image Fusion
• Adverse Weather
• Feature Restoration
• Cross-modal Interaction
• Image Degradation

Innovation

• mask-guided fusion
• pseudo ground truth
• cross-modal attention
• adverse weather image fusion
• feature restoration