Reinforced Diffuser for Red Teaming Large Vision-Language Models

2025-03-08

AI Summary

This work identifies an alignment blind spot in large vision-language models (VLMs): existing safety alignment mechanisms fail to suppress image-induced harmful text generation in toxic text continuation tasks. To address this, we propose Red-Teaming Diffusion (RTD), a two-stage framework: (1) LLM-guided prompt search to identify vulnerable inputs, and (2) diffusion-based adversarial fine-tuning to synthesize highly toxic counterfactual images. RTD integrates a toxicity scorer with alignment reward modeling to enable end-to-end red-teaming optimization. Experiments show RTD increases toxicity triggering rates by 10.69% on the original test set and 8.91% on the held-out set for LLaVA. Moreover, RTD exhibits strong cross-model transferability—boosting toxicity by +5.1% on Gemini and +26.83% on LLaMA-3. This is the first systematic demonstration of alignment failure in VLM text continuation scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid advancement of large Vision-Language Models (VLMs) has raised significant safety concerns, particularly regarding their vulnerability to jailbreak attacks. While existing research primarily focuses on VLMs' susceptibility to harmful instructions, this work identifies a critical yet overlooked vulnerability: current alignment mechanisms often fail to address the risks posed by toxic text continuation tasks. To investigate this issue, we propose a novel Red Team Diffuser (RTD) framework, which leverages reinforcement learning to generate red team images that effectively induce highly toxic continuations from target black-box VLMs. The RTD pipeline begins with a greedy search for high-quality image prompts that maximize the toxicity of VLM-generated sentence continuations, guided by a Large Language Model (LLM). These prompts are then used as input for the reinforcement fine-tuning of a diffusion model, which employs toxicity and alignment rewards to further amplify harmful outputs. Experimental results demonstrate the effectiveness of RTD, increasing the toxicity rate of LLaVA outputs by 10.69% on the original attack set and 8.91% on a hold-out set. Moreover, RTD exhibits strong cross-model transferability, raising the toxicity rate by 5.1% on Gemini and 26.83% on LLaMA. These findings reveal significant deficiencies in existing alignment strategies, particularly their inability to prevent harmful continuations. Our work underscores the urgent need for more robust and adaptive alignment mechanisms to ensure the safe deployment of VLMs in real-world applications.

Problem

• Identifies vulnerability in VLM alignment mechanisms
• Proposes RTD framework to induce toxic continuations
• Highlights need for robust alignment strategies

Innovation

• Reinforcement learning for toxic image generation
• Greedy search optimizes image prompts for toxicity
• Diffusion model fine-tuned with toxicity rewards