ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning

2023-12-13

International Journal of Computer Vision

AI Summary

Existing image fusion methods rely on handcrafted loss functions and lack a general, learnable, reference-free supervision mechanism, limiting their cross-task generalization capability. To address this, we propose ReFusion—a novel meta-learning-driven framework that generates task-specific losses by reconstructing target images from source inputs. ReFusion introduces a differentiable, parameterized loss network jointly optimized with the fusion model and a reconstruction module, enabling adaptive, reference-free loss generation without ground-truth supervision. This eliminates manual loss design, achieving fully end-to-end trainable, task-adaptive loss functions. Extensive experiments demonstrate state-of-the-art performance across diverse fusion tasks—including infrared–visible, medical, multi-focus, and multi-exposure fusion—while significantly improving cross-modal and cross-task generalization.

Abstract

Image fusion aims to combine information from multiple source images into a single one with more comprehensive informational content. The significant challenges for deep learning-based image fusion algorithms are the lack of a definitive ground truth as well as the corresponding distance measurement, with current manually given loss functions constrain the flexibility of model and generalizability for unified fusion tasks. To overcome these limitations, we introduce a unified image fusion framework based on meta-learning, named ReFusion, which provides a learning paradigm that obtains the optimal fusion loss for various fusion tasks based on reconstructing the source images. Compared to existing methods, ReFusion employs a parameterized loss function, dynamically adjusted by the training framework according to the specific scenario and task. ReFusion is constituted by three components: a fusion module, a loss proposal module, and a source reconstruction module. To ensure the fusion module maximally preserves the information from the source images, enabling the reconstruction of the source images from the fused image, we adopt a meta-learning strategy to train the loss proposal module using reconstruction loss. The update of the fusion module relies on the fusion loss proposed by the loss proposal module. The alternating updates of the three modules mutually facilitate each other, aiming to propose an appropriate fusion loss for different tasks and yield satisfactory fusion results. Extensive experiments demonstrate that ReFusion is capable of adapting to various tasks, including infrared-visible, medical, multi-focus, and multi-exposure image fusion. The code will be released.

Problem

• Image Fusion
• Deep Learning
• Evaluation Metrics

Innovation

• ReFusion Framework
• Meta-learning
• Adaptive Image Fusion