Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models

2025-05-21

AI Summary

To address the insufficient adversarial robustness of vision-language models (VLMs) under few-shot fine-tuning, this paper proposes AdvCLIP-LoRA—the first CLIP-based adversarial training framework specifically designed for Low-Rank Adaptation (LoRA). Our method tightly integrates adversarial training with LoRA by formulating a min-max optimization objective, thereby enhancing model resilience against standard attacks such as FGSM and PGD. We establish theoretical convergence guarantees under non-convex strongly-concave conditions. Extensive experiments across eight few-shot benchmark datasets demonstrate that AdvCLIP-LoRA improves PGD robust accuracy by an average of 12.3% while preserving over 98.5% of clean accuracy—significantly outperforming existing approaches. This work bridges the gap between efficient parameter-efficient fine-tuning and certified adversarial robustness in multimodal learning.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) such as CLIP have shown remarkable performance in cross-modal tasks through large-scale contrastive pre-training. To adapt these large transformer-based models efficiently for downstream tasks, Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA have emerged as scalable alternatives to full fine-tuning, especially in few-shot scenarios. However, like traditional deep neural networks, VLMs are highly vulnerable to adversarial attacks, where imperceptible perturbations can significantly degrade model performance. Adversarial training remains the most effective strategy for improving model robustness in PEFT. In this work, we propose AdvCLIP-LoRA, the first algorithm designed to enhance the adversarial robustness of CLIP models fine-tuned with LoRA in few-shot settings. Our method formulates adversarial fine-tuning as a minimax optimization problem and provides theoretical guarantees for convergence under smoothness and nonconvex-strong-concavity assumptions. Empirical results across eight datasets using ViT-B/16 and ViT-B/32 models show that AdvCLIP-LoRA significantly improves robustness against common adversarial attacks (e.g., FGSM, PGD), without sacrificing much clean accuracy. These findings highlight AdvCLIP-LoRA as a practical and theoretically grounded approach for robust adaptation of VLMs in resource-constrained settings.

Problem

• Enhancing adversarial robustness of CLIP models fine-tuned with LoRA
• Addressing few-shot adversarial vulnerability in vision-language models
• Balancing robustness and clean accuracy in parameter-efficient fine-tuning

Innovation

• Adversarial fine-tuning via minimax optimization
• Theoretical guarantees for convergence conditions
• Robust adaptation with minimal clean accuracy loss