🤖 AI Summary
Multimodal image fusion faces two key challenges: gradient conflicts arising from cross-modal parameter sharing, which degrade performance; and modality-specific encoders that improve fusion quality yet harm task generalization. To address these, we propose a unified fusion framework integrating three novel components: semantic-aware channel pruning (to retain discriminative features), geometric affine modulation (to model inter-modal spatial discrepancies), and text-guided channel perturbation (to inject semantic priors and enhance robustness). Our method synergistically leverages pretrained semantic knowledge, channel-level perturbations, and affine transformations—achieving selective feature learning and strong cross-task generalization without introducing modality-specific parameters. Extensive experiments demonstrate consistent and significant improvements over state-of-the-art methods on major fusion benchmarks and downstream detection and segmentation tasks.
📝 Abstract
Multi-modality image fusion enhances scene perception by combining complementary information. Unified models aim to share parameters across modalities for multi-modality image fusion, but large modality differences often cause gradient conflicts, limiting performance. Some methods introduce modality-specific encoders to enhance feature perception and improve fusion quality. However, this strategy reduces generalisation across different fusion tasks. To overcome this limitation, we propose a unified multi-modality image fusion framework based on channel perturbation and pre-trained knowledge integration (UP-Fusion). To suppress redundant modal information and emphasize key features, we propose the Semantic-Aware Channel Pruning Module (SCPM), which leverages the semantic perception capability of a pre-trained model to filter and enhance multi-modality feature channels. Furthermore, we proposed the Geometric Affine Modulation Module (GAM), which uses original modal features to apply affine transformations on initial fusion features to maintain the feature encoder modal discriminability. Finally, we apply a Text-Guided Channel Perturbation Module (TCPM) during decoding to reshape the channel distribution, reducing the dependence on modality-specific channels. Extensive experiments demonstrate that the proposed algorithm outperforms existing methods on both multi-modality image fusion and downstream tasks.