Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models

2024-07-02

Conference on Empirical Methods in Natural Language Processing

AI Summary

Medical vision-language models suffer significant performance degradation when fine-tuned on upstream data corrupted by adversarial noise, further exacerbated by data scarcity, label noise, and privacy constraints. To address this, we propose RAN, a lightweight robust fine-tuning framework. We first empirically identify that moderate-strength adversarial noise—contrary to conventional assumptions—enhances both model robustness and cross-task transferability. RAN mitigates noise-induced distortions via three synergistic components: noise-aware fine-tuning, contrastive learning-driven feature disentanglement, and dynamic re-alignment—all without increasing inference overhead. Evaluated on MedMNIST-VL and PathVLM benchmarks, RAN improves downstream classification and retrieval accuracy by 3.2–5.7% over strong baselines and substantially outperforms mainstream adapter methods (e.g., LoRA, Adapter) in adversarial robustness, demonstrating superior noise resilience under realistic clinical data constraints.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Fine-tuning pre-trained Vision-Language Models (VLMs) has shown remarkable capabilities in medical image and textual depiction synergy. Nevertheless, many pre-training datasets are restricted by patient privacy concerns, potentially containing noise that can adversely affect downstream performance. Moreover, the growing reliance on multi-modal generation exacerbates this issue because of its susceptibility to adversarial attacks. To investigate how VLMs trained on adversarial noisy data perform on downstream medical tasks, we first craft noisy upstream datasets using multi-modal adversarial attacks. Through our comprehensive analysis, we unveil that moderate noise enhances model robustness and transferability, but increasing noise levels negatively impact downstream task performance. To mitigate this issue, we propose rectify adversarial noise (RAN) framework, a recipe designed to effectively defend adversarial attacks and rectify the influence of upstream noise during fine-tuning.

Problem

• Medical Image Protection
• Model Robustness
• Security in Machine Learning

Innovation

• Adversarial Noise Correction
• Medical Image and Text Models
• Robustness Enhancement