Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks

2025-04-02

AI Summary

Vision-language models (VLMs) are vulnerable to jailbreaking attacks under simple image perturbations such as Gaussian noise, primarily due to the lack of noise-robustness modeling in prevailing training paradigms. Method: This work identifies the absence of noise-aware training as a fundamental cause of VLM security vulnerabilities and introduces Robust-VLGuard—a novel multimodal safety dataset—alongside DiffPure-VLM, the first end-to-end defense framework leveraging diffusion-based reverse purification. It integrates noise-augmented fine-tuning, adversarial perturbation distribution transfer, and construction of multimodal aligned/misaligned data. Contribution/Results: Experiments demonstrate that DiffPure-VLM substantially reduces success rates across diverse optimization-based visual perturbation attacks. It maintains semantic understanding across varying noise intensities and achieves simultaneous improvements in both robustness and accuracy on standard benchmarks.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Vision-Language Models (VLMs) extend the capabilities of Large Language Models (LLMs) by incorporating visual information, yet they remain vulnerable to jailbreak attacks, especially when processing noisy or corrupted images. Although existing VLMs adopt security measures during training to mitigate such attacks, vulnerabilities associated with noise-augmented visual inputs are overlooked. In this work, we identify that missing noise-augmented training causes critical security gaps: many VLMs are susceptible to even simple perturbations such as Gaussian noise. To address this challenge, we propose Robust-VLGuard, a multimodal safety dataset with aligned / misaligned image-text pairs, combined with noise-augmented fine-tuning that reduces attack success rates while preserving functionality of VLM. For stronger optimization-based visual perturbation attacks, we propose DiffPure-VLM, leveraging diffusion models to convert adversarial perturbations into Gaussian-like noise, which can be defended by VLMs with noise-augmented safety fine-tuning. Experimental results demonstrate that the distribution-shifting property of diffusion model aligns well with our fine-tuned VLMs, significantly mitigating adversarial perturbations across varying intensities. The dataset and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.

Problem

• VLMs vulnerable to Gaussian noise attacks
• Lack noise-augmented training for security
• Need defense against adversarial visual perturbations

Innovation

• Noise-augmented fine-tuning for VLMs
• Diffusion models for adversarial noise conversion
• Multimodal safety dataset with aligned pairs