AdvLoRA: Adversarial Low-Rank Adaptation of Vision-Language Models

2024-04-20

arXiv.org

AI Summary

Visual-language models (VLMs) exhibit security vulnerabilities and prohibitive computational overhead during adversarial adaptation. This paper first identifies an intrinsic low-rank structure in their adversarial update process and proposes LoRA-Adv—a parameter-efficient, robust low-rank adversarial adaptation method. LoRA-Adv reparameterizes adversarial fine-tuning by integrating parameter clustering, gradient alignment, and adaptive updating strategies. Evaluated across multiple VLM benchmarks, LoRA-Adv achieves robustness comparable to full-parameter adversarial fine-tuning while using less than 10% of the trainable parameters and only 25% of the training cost. It significantly mitigates backdoor attacks and adversarial example threats. By unifying efficiency and security, LoRA-Adv establishes a novel paradigm for safe and computationally efficient adversarial adaptation of VLMs.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) play a crucial role in the advancement of Artificial General Intelligence (AGI). As AGI rapidly evolves, addressing security concerns has emerged as one of the most significant challenges for VLMs. In this paper, we present extensive experiments that expose the vulnerabilities of conventional adaptation methods for VLMs, highlighting significant security risks. Moreover, as VLMs grow in size, the application of traditional adversarial adaptation techniques incurs substantial computational costs. To address these issues, we propose a parameter-efficient adversarial adaptation method called extbf{ extit{AdvLoRA}} based on Low-Rank Adaptation. We investigate and reveal the inherent low-rank properties involved in adversarial adaptation for VLMs. Different from LoRA, we enhance the efficiency and robustness of adversarial adaptation by introducing a novel reparameterization method that leverages parameter clustering and alignment. Additionally, we propose an adaptive parameter update strategy to further bolster robustness. These innovations enable our AdvLoRA to mitigate issues related to model security and resource wastage. Extensive experiments confirm the effectiveness and efficiency of AdvLoRA.

Problem

• Addressing security vulnerabilities in VLMs
• Reducing computational costs in adversarial adaptation
• Enhancing efficiency and robustness of VLMs

Innovation

• Low-Rank Adaptation
• Parameter clustering and alignment
• Adaptive parameter update