Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks

2024-07-30

European Conference on Computer Vision

AI Summary

To address insufficient transferability in cross-model, cross-domain black-box adversarial attacks, this paper proposes PDCL-Attack—the first framework integrating prompt learning into generative adversarial attacks. Leveraging CLIP’s vision-language joint representation capability, PDCL-Attack employs text-prompt-driven contrastive learning to achieve semantically aligned feature guidance, substantially improving the generalizability of generated perturbations across unseen models and data distributions. Its core innovation lies in formulating transferability as a cross-modal semantic consistency optimization problem, departing from conventional pixel-level perturbation optimization. Under a multi-source white-box training and black-box evaluation setting targeting unknown models and domains, PDCL-Attack achieves an average 12.6% higher transfer success rate than state-of-the-art methods. Moreover, it is the first work to empirically validate both the effectiveness and scalability of prompt learning in generative adversarial attacks.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent vision-language foundation models, such as CLIP, have demonstrated superior capabilities in learning representations that can be transferable across diverse range of downstream tasks and domains. With the emergence of such powerful models, it has become crucial to effectively leverage their capabilities in tackling challenging vision tasks. On the other hand, only a few works have focused on devising adversarial examples that transfer well to both unknown domains and model architectures. In this paper, we propose a novel transfer attack method called PDCL-Attack, which leverages the CLIP model to enhance the transferability of adversarial perturbations generated by a generative model-based attack framework. Specifically, we formulate an effective prompt-driven feature guidance by harnessing the semantic representation power of text, particularly from the ground-truth class labels of input images. To the best of our knowledge, we are the first to introduce prompt learning to enhance the transferable generative attacks. Extensive experiments conducted across various cross-domain and cross-model settings empirically validate our approach, demonstrating its superiority over state-of-the-art methods.

Problem

• Enhancing adversarial attack transferability across domains and models.
• Leveraging CLIP model for effective prompt-driven feature guidance.
• Introducing prompt learning to improve generative adversarial attacks.

Innovation

• Uses CLIP model for transferable adversarial attacks
• Introduces prompt-driven feature guidance with text semantics
• First to apply prompt learning in generative attacks