AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models

2024-10-07

AI Summary

Existing vision-language models (VLMs) are vulnerable to image-based adversarial attacks; however, conventional targeted attacks rely on prior class labels, limiting their generalizability and practical applicability. To address this, we propose AnyAttack—the first large-scale self-supervised adversarial attack framework that requires neither ground-truth labels nor target categories. It leverages unsupervised pretraining on LAION-400M and contrastive learning–driven visual-language alignment to optimize label-agnostic, gradient-free universal perturbations. AnyAttack introduces a novel foundation-model–based self-supervised adversarial pretraining paradigm, enabling zero-shot black-box transfer attacks across diverse VLMs—including open-source and commercial systems (e.g., Gemini, GPT, Claude, Copilot)—as well as across models and tasks. Evaluated on five open-source VLMs, it achieves an average attack success rate exceeding 82%, and successfully exposes systematic security vulnerabilities in leading commercial VLMs.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Due to their multimodal capabilities, Vision-Language Models (VLMs) have found numerous impactful applications in real-world scenarios. However, recent studies have revealed that VLMs are vulnerable to image-based adversarial attacks. Traditional targeted adversarial attacks require specific targets and labels, limiting their real-world impact.We present AnyAttack, a self-supervised framework that transcends the limitations of conventional attacks through a novel foundation model approach. By pre-training on the massive LAION-400M dataset without label supervision, AnyAttack achieves unprecedented flexibility - enabling any image to be transformed into an attack vector targeting any desired output across different VLMs.This approach fundamentally changes the threat landscape, making adversarial capabilities accessible at an unprecedented scale. Our extensive validation across five open-source VLMs (CLIP, BLIP, BLIP2, InstructBLIP, and MiniGPT-4) demonstrates AnyAttack's effectiveness across diverse multimodal tasks. Most concerning, AnyAttack seamlessly transfers to commercial systems including Google Gemini, Claude Sonnet, Microsoft Copilot and OpenAI GPT, revealing a systemic vulnerability requiring immediate attention.

Problem

• Self-supervised adversarial attacks on vision-language models
• Large-scale vulnerability across diverse multimodal tasks
• Transferability to commercial systems without label supervision

Innovation

• Self-supervised adversarial attack framework
• Pre-trained on LAION-400M without labels
• Transfers to commercial VLMs effectively