Task-Agnostic Attacks Against Vision Foundation Models

2025-03-05

AI Summary

This work introduces a task-agnostic universal adversarial attack against Vision Foundation Models (VFMs) to expose their feature-level security vulnerabilities across tasks and models. Methodologically, it bypasses downstream labels or task-specific heads, instead directly optimizing gradient-based perturbations in the feature space to maximally degrade the model’s general representation capability. It establishes the first unified adversarial framework tailored for VFMs, breaking from conventional task-specific attack paradigms. Evaluated on prominent VFMs—including CLIP, DINOv2, and SAM—the attack consistently degrades performance across diverse downstream tasks (e.g., classification, segmentation, depth estimation, retrieval, and VQA), yielding an average accuracy drop of 32.7%. Crucially, it exhibits strong cross-model transferability. The core contribution lies in formally characterizing the security boundary of universal visual representations in VFMs and providing a novel paradigm for robustness evaluation and defense.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

The study of security in machine learning mainly focuses on downstream task-specific attacks, where the adversarial example is obtained by optimizing a loss function specific to the downstream task. At the same time, it has become standard practice for machine learning practitioners to adopt publicly available pre-trained vision foundation models, effectively sharing a common backbone architecture across a multitude of applications such as classification, segmentation, depth estimation, retrieval, question-answering and more. The study of attacks on such foundation models and their impact to multiple downstream tasks remains vastly unexplored. This work proposes a general framework that forges task-agnostic adversarial examples by maximally disrupting the feature representation obtained with foundation models. We extensively evaluate the security of the feature representations obtained by popular vision foundation models by measuring the impact of this attack on multiple downstream tasks and its transferability between models.

Problem

• Explores task-agnostic attacks on vision foundation models.
• Assesses impact on multiple downstream tasks and model transferability.
• Develops framework for adversarial examples disrupting feature representations.

Innovation

• Task-gnostic adversarial examples generation
• Maximally disrupts foundation model features
• Evaluates attack impact across multiple tasks