HyPSAM: Hybrid Prompt-driven Segment Anything Model for RGB-Thermal Salient Object Detection

2025-09-23

AI Summary

RGB-T salient object detection faces dual challenges: insufficient cross-modal feature fusion and scarce annotated data, leading to ambiguous object boundaries and poor segmentation completeness. To address these, we propose HyPSAM—a hybrid prompt-driven “Segment Anything” model. HyPSAM introduces a Dynamic Fusion Network (DFNet) for adaptive RGB-thermal feature interaction, and a plug-and-play Prompt-to-Refinement Network (P2RNet) that jointly leverages text, mask, and bounding-box prompts to guide SAM for fine-grained segmentation. Additionally, dynamic convolution and multi-branch decoding are incorporated to enhance boundary localization accuracy. Crucially, HyPSAM requires no fine-tuning of SAM’s backbone, ensuring zero-shot transferability and strong generalizability. Evaluated on three benchmark RGB-T datasets, HyPSAM achieves state-of-the-art performance. Moreover, it serves as a plug-in module that consistently improves existing methods, empirically validating the effectiveness of multimodal hybrid prompting for salient object segmentation.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

RGB-thermal salient object detection (RGB-T SOD) aims to identify prominent objects by integrating complementary information from RGB and thermal modalities. However, learning the precise boundaries and complete objects remains challenging due to the intrinsic insufficient feature fusion and the extrinsic limitations of data scarcity. In this paper, we propose a novel hybrid prompt-driven segment anything model (HyPSAM), which leverages the zero-shot generalization capabilities of the segment anything model (SAM) for RGB-T SOD. Specifically, we first propose a dynamic fusion network (DFNet) that generates high-quality initial saliency maps as visual prompts. DFNet employs dynamic convolution and multi-branch decoding to facilitate adaptive cross-modality interaction, overcoming the limitations of fixed-parameter kernels and enhancing multi-modal feature representation. Moreover, we propose a plug-and-play refinement network (P2RNet), which serves as a general optimization strategy to guide SAM in refining saliency maps by using hybrid prompts. The text prompt ensures reliable modality input, while the mask and box prompts enable precise salient object localization. Extensive experiments on three public datasets demonstrate that our method achieves state-of-the-art performance. Notably, HyPSAM has remarkable versatility, seamlessly integrating with different RGB-T SOD methods to achieve significant performance gains, thereby highlighting the potential of prompt engineering in this field. The code and results of our method are available at: https://github.com/milotic233/HyPSAM.

Problem

• Improving RGB-thermal salient object detection with precise boundaries
• Overcoming insufficient feature fusion between RGB and thermal modalities
• Addressing data scarcity limitations in multi-modal object detection

Innovation

• Hybrid prompt-driven SAM for RGB-thermal detection
• Dynamic fusion network for adaptive cross-modality interaction
• Plug-and-play refinement network using hybrid prompts