Task-driven Image Fusion with Learnable Fusion Loss

2024-12-04

arXiv.org

AI Summary

Existing image fusion methods rely on predefined fusion losses, limiting their adaptability to diverse downstream tasks and thus compromising generalizability and flexibility. To address this, we propose TDFusion, a task-driven learnable fusion framework that employs only the loss of a downstream task (e.g., segmentation or detection) as supervision, dynamically generating a task-specific fusion loss. Our key contributions are twofold: (i) the first integration of meta-learning into fusion loss design, enabling a parameterized, learnable loss generation module; and (ii) a dual-module iterative optimization scheme that facilitates end-to-end co-training of the fusion network and downstream task model. Extensive experiments across four benchmark fusion datasets and multiple downstream tasks demonstrate that TDFusion consistently outperforms state-of-the-art methods, validating its architecture-agnostic applicability and effectiveness for arbitrary fusion and task networks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Multi-modal image fusion aggregates information from multiple sensor sources, achieving superior visual quality and perceptual features compared to single-source images, often improving downstream tasks. However, current fusion methods for downstream tasks still use predefined fusion objectives that potentially mismatch the downstream tasks, limiting adaptive guidance and reducing model flexibility. To address this, we propose Task-driven Image Fusion (TDFusion), a fusion framework incorporating a learnable fusion loss guided by task loss. Specifically, our fusion loss includes learnable parameters modeled by a neural network called the loss generation module. This module is supervised by the downstream task loss in a meta-learning manner. The learning objective is to minimize the task loss of fused images after optimizing the fusion module with the fusion loss. Iterative updates between the fusion module and the loss module ensure that the fusion network evolves toward minimizing task loss, guiding the fusion process toward the task objectives. TDFusion's training relies entirely on the downstream task loss, making it adaptable to any specific task. It can be applied to any architecture of fusion and task networks. Experiments demonstrate TDFusion's performance through fusion experiments conducted on four different datasets, in addition to evaluations on semantic segmentation and object detection tasks.

Problem

• Mismatch between predefined fusion objectives and downstream tasks
• Lack of adaptive guidance in current fusion methods
• Limited model flexibility for task-specific image fusion

Innovation

• Learnable fusion loss guided by task loss
• Meta-learning supervised loss generation module
• Iterative updates between fusion and loss modules