Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection

2026-07-11

AI Summary

This work addresses the vulnerability of vision-language models (VLMs) to textual query attacks that can inadvertently expose sensitive information in user images. Existing reversible defense methods struggle to simultaneously preserve privacy, visual fidelity, and perfect invertibility in multimodal settings. To overcome this challenge, we propose CloakDiff, the first framework enabling high-fidelity, imperceptible, and fully reversible privacy protection against VLM-based attacks. CloakDiff achieves this by jointly perturbing pixel-space embeddings and cross-attention maps through diffusion-guided adversarial editing, integrated with invertible neural networks and an EDM-inspired sampling strategy. Extensive experiments demonstrate that CloakDiff effectively mitigates textual query attacks across diverse datasets and VLM architectures while maintaining high image quality and lossless recoverability, and further exhibits strong transfer robustness across models and prompts.

Abstract

Vision Language Models (VLMs) offer powerful multimodal ability but also expose users to text-based privacy attacks where adversaries crawl online photos and query VLMs to extract sensitive attributes. Existing reversible adversarial example (RAE) methods protect images in purely visual tasks but fail in multimodal settings, and current adversarial examples on VLMs rely on high frequency noise that severely degrades visual quality. We propose CloakDiff, the first framework for reversible, high fidelity privacy protection against text-based query attacks in VLMs. CloakDiff produces imperceptible adversarial examples by combining diffusion based adversarial editing with an invertible network that embeds the original image for lossless recovery. It perturbs both pixel space embeddings and manipulates latent cross attention maps to ensure strong cross-model and cross-prompt transferability while preserving global visual structure. To further enhance fidelity, we design EDM Heuristic Sampling, a principled diffusion schedule for adversarial guidance. Experiments on multiple datasets and VLMs demonstrate that CloakDiff delivers multimodal privacy preservation with high visual quality and reversibility.

Problem

• Vision-Language Models
• Privacy Protection
• Adversarial Examples
• Reversibility
• Imperceptibility

Innovation

• reversible adversarial examples
• vision-language models
• diffusion-based editing
• invertible network
• privacy protection