TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion

2025-06-20

AI Summary

To address insufficient semantic information utilization and suboptimal downstream task performance in infrared and visible light image fusion (IVF), this paper introduces large vision-language models (VLMs) into IVF for the first time, proposing a hierarchical text-semantic-guided fusion paradigm. Specifically, mask-guided cross-attention (MGCA) is devised to model pixel-level semantic alignment, while text-driven gated attention (TDAF) enables high-level semantic modulation. By jointly optimizing feature fusion under dual semantic guidance—both mask-level and text-level—the method significantly enhances the utility of fused images for downstream tasks such as object detection and semantic segmentation. Extensive experiments demonstrate consistent superiority over existing state-of-the-art methods across multiple benchmarks, validating the effectiveness of integrating multimodal linguistic priors into IVF.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities, producing more informative and comprehensive outputs. Recently, text-guided IVF has shown great potential due to its flexibility and versatility. However, the effective integration and utilization of textual semantic information remains insufficiently studied. To tackle these challenges, we introduce textual semantics at two levels: the mask semantic level and the text semantic level, both derived from textual descriptions extracted by large Vision-Language Models (VLMs). Building on this, we propose Textual Semantic Guidance for infrared and visible image fusion, termed TeSG, which guides the image synthesis process in a way that is optimized for downstream tasks such as detection and segmentation. Specifically, TeSG consists of three core components: a Semantic Information Generator (SIG), a Mask-Guided Cross-Attention (MGCA) module, and a Text-Driven Attentional Fusion (TDAF) module. The SIG generates mask and text semantics based on textual descriptions. The MGCA module performs initial attention-based fusion of visual features from both infrared and visible images, guided by mask semantics. Finally, the TDAF module refines the fusion process with gated attention driven by text semantics. Extensive experiments demonstrate the competitiveness of our approach, particularly in terms of performance on downstream tasks, compared to existing state-of-the-art methods.

Problem

• Effective integration of textual semantics in image fusion
• Optimizing fusion for downstream tasks like detection
• Leveraging vision-language models for semantic guidance

Innovation

• Uses Vision-Language Models for text semantics
• Integrates mask and text semantic guidance
• Employs gated attention for refined fusion