Backbone is All You Need: Assessing Vulnerabilities of Frozen Foundation Models in Synthetic Image Forensics

2026-05-13

AI Summary

This work reveals a critical security vulnerability in synthetic image detection systems that rely on frozen pre-trained Vision Transformer (ViT) backbones: effective attacks can be mounted using only knowledge of the backbone itself. The authors propose SIAA, a gray-box adversarial attack that iteratively crafts perturbations in the ViT feature space of a target detector without requiring access to the full model. This approach demonstrates for the first time that near-white-box attack performance is achievable solely with frozen backbone information, thereby challenging the prevailing assumption that such backbones provide reliable forensic signals. Extensive experiments show that SIAA achieves high attack success rates across diverse ViT-based detectors and gray-box settings, while also exhibiting strong cross-model transferability, exposing the fragility of current detection frameworks under adversarial conditions.

Abstract

As AI-generated synthetic images become increasingly realistic, Vision Transformers (ViTs) have emerged as a cornerstone of modern deepfake detection. However, the prevailing reliance on frozen, pre-trained backbones introduces a subtle yet critical vulnerability. In this work, we present the Surrogate Iterative Adversarial Attack (SIAA), a gray-box attack that exploits knowledge of the detector's ViT backbone alone and operates entirely within the target detector's feature space to craft highly effective adversarial examples. Through our experiments, involving multiple ViT-based detectors and diverse gray-box scenarios, including few-shot learning, complete training misalignment and attack transferability tests, we demonstrate that this vulnerability consistently yields high attack success rates, often approaching white-box performance. By doing so, we reveal that backbone knowledge alone is sufficient to undermine detector reliability, highlighting the urgent need for more resilient defenses in adversarial multimedia forensics.

Problem

• synthetic image forensics
• frozen foundation models
• Vision Transformers
• adversarial vulnerability
• deepfake detection

Innovation

• Surrogate Iterative Adversarial Attack
• Vision Transformers
• frozen foundation models
• synthetic image forensics
• gray-box attack