Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT

2026-07-08

AI Summary

Existing attention-based defense methods for Vision Transformers are vulnerable to localized adversarial attacks. This work proposes an “adversarial decoy” approach that, for the first time, decouples the attack objective from defense evasion by generating image patches—optimized independently of the original attack—that redirect the model’s attention away from adversarial regions. By steering high attention scores toward non-adversarial areas, the method effectively misleads defenses that rely on attention magnitude to identify perturbations. A hierarchical optimization strategy is employed to enhance attention on target tokens, making the approach compatible with diverse ViT architectures and adversarial patch attacks. Experiments on ImageNet demonstrate that the method successfully shifts high attention away from genuine adversarial regions while maintaining high attack success rates, thereby exposing a fundamental limitation in using attention magnitude alone as an indicator of adversarial relevance.

Abstract

Vision Transformers (ViTs) remain vulnerable to localized adversarial attacks, e.g., adversarial patches, while recent test-time defenses mitigate them by suppressing image tokens with abnormally high attention scores. These defenses exploit a strong coupling between attention and adversarial effectiveness: adversarial tokens often need to attract substantial attention to influence the prediction. We introduce adversarial decoys, independently optimized image patches that redirect the attention, and therefore related defenses, toward selected target tokens. Rather than jointly optimizing misclassifications and defense evasion, our approach decouples the two objectives: the original adversarial region induces the incorrect prediction, while a separate decoy manipulates the attention ranking used by the defense. A layer-wise objective increases target-token attention and promotes these tokens above competing non-target ones. Since the decoy is optimized independently of the underlying attack, the method is attack-agnostic and can be easily integrated with any existing adversarial patch attack. Experiments on ImageNet across multiple ViT architectures and attacks show that decoys can redirect high attention scores away from the true adversarial region while preserving much of the attack effectiveness. These results reveal a fundamental limitation of using attention magnitude as an indicator of adversarial relevance.

Problem

• adversarial attacks
• attention-based defenses
• Vision Transformers
• adversarial patches
• defense evasion

Innovation

• adversarial decoys
• attention manipulation
• vision transformers
• test-time defense
• attack-agnostic