Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models

2026-04-13

AI Summary

This work addresses the vulnerability of vision-language models such as CLIP to adversarial attacks in zero-shot settings, where existing fine-tuning approaches often fail to preserve robustness and degrade original performance due to mismatches with the pretraining data distribution and objective. To overcome these limitations, the authors propose AdvFLYP, the first method to replicate CLIP’s pretraining pipeline during adversarial fine-tuning. AdvFLYP leverages web-crawled image–text pairs to generate adversarial images and aligns their textual embeddings via contrastive loss, while incorporating both feature-level and logit-level regularization to mitigate embedding distortions caused by adversarial perturbations. Evaluated across 14 cross-domain downstream tasks, AdvFLYP substantially outperforms prior methods, simultaneously enhancing adversarial robustness, preserving or even improving clean accuracy, and maintaining strong zero-shot transfer capabilities.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite their impressive zero-shot abilities, vision-language models such as CLIP have been shown to be susceptible to adversarial attacks. To enhance its adversarial robustness, recent studies finetune the pretrained vision encoder of CLIP with adversarial examples on a proxy dataset such as ImageNet by aligning adversarial images with correct class labels. However, these methods overlook the important roles of training data distributions and learning objectives, resulting in reduced zero-shot capabilities and limited transferability of robustness across domains and datasets. In this work, we propose a simple yet effective paradigm AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model. Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss. To alleviate distortion of adversarial image embeddings of noisy web images, we further propose to regularise AdvFLYP by penalising deviation of adversarial image features. We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively. Extensive experiments on 14 downstream datasets spanning various domains show the superiority of our paradigm over mainstream practices. Our code and model weights are released at https://github.com/Sxing2/AdvFLYP.

Problem

• adversarial robustness
• vision-language models
• zero-shot learning
• adversarial fine-tuning
• transferability

Innovation

• adversarial robustness
• vision-language models
• contrastive learning
• adversarial fine-tuning
• zero-shot transfer