BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming

2026-06-23

AI Summary

This work addresses key challenges in transferring pretrained vision-language models to biomedical imaging—namely data scarcity, subtle inter-class differences, and the neglect of confusable negative samples. To overcome these limitations, the authors propose BioMedVR, a novel framework that introduces visual reprogramming to biomedical imaging for the first time. It employs a compact, learnable module for efficient few-shot adaptation and innovatively leverages large language models to generate confusion-aware attributes, which inform a newly designed confusion-suppression loss. Furthermore, BioMedVR integrates a mixture-of-prompts-experts architecture with an adaptive gating mechanism to jointly optimize primary class recognition and confusion elimination. Evaluated across 18 datasets—including 11 biomedical benchmarks—the method significantly outperforms existing approaches, demonstrating superior accuracy and generalization capability.

Abstract

Recent advances in vision-language models (VLMs) such as CLIP have demonstrated strong generalization across natural-image domains. However, adapting these models to biomedical imaging is non-trivial: full-model fine-tuning is computationally expensive, while medical data are often scarce and exhibit subtle, fine-grained inter-class differences, making parameter-efficient adaptation particularly critical. Visual Reprogramming (VR) offers a parameter-efficient alternative by injecting learnable perturbations into the input space, but existing VR approaches for VLMs mainly focus on positive class prompts and overlook confusing negatives, leading to miscalibrated predictions in fine-grained medical scenarios. We present BioMedVR, the first VR-based framework for biomedical imaging, enabling few-shot adaptation of pretrained VLMs through compact learnable VR modules. To mitigate class confusion, we introduce a Confusion Minimization Mechanism that leverages LLM-generated confusion-aware attributes together with a Confusion-Suppression Loss to explicitly reduce false-positive alignment. Moreover, the designed Mixture-of-Prompt Experts combines a positive expert for main-class discrimination and a negative expert for confusion suppression, balanced via adaptive gating. Extensive experiments on 18 datasets, including 11 biomedical datasets and 7 natural image benchmarks, demonstrate that BioMedVR achieves superior accuracy and generalization, effectively bridging VR and VLMs in biomedical domains.

Problem

• biomedical imaging
• vision-language models
• visual reprogramming
• class confusion
• parameter-efficient adaptation

Innovation

• Visual Reprogramming
• Vision-Language Models
• Confusion-Aware Learning
• Mixture-of-Prompt Experts
• Biomedical Imaging