Adversarial Attention Perturbations for Large Object Detection Transformers

2025-08-04

AI Summary

Existing adversarial perturbation methods struggle to simultaneously attack both CNN- and Transformer-based object detectors, particularly exhibiting limited efficacy against large-scale Transformer models. Method: We propose a unified adversarial attention perturbation framework—the first to enable efficient and stealthy attacks against twelve state-of-the-art COCO benchmark detectors spanning both architectures. Our approach introduces a learnable attention mechanism to dynamically localize model-vulnerable regions, coupled with dual-feature loss and multi-stage gradient optimization to iteratively inject imperceptible perturbations under strict visual invisibility constraints. Contribution/Results: Experiments demonstrate up to an 83% improvement in attack success rate and a 30.6% gain in perturbation effectiveness, while maintaining high stealthiness and computational efficiency. The framework establishes a novel paradigm for robustness evaluation of large vision models, offering broad applicability across heterogeneous detector architectures.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to attacking CNN-based detectors or weak against transformer-based detectors. This paper presents an Attention-Focused Offensive Gradient (AFOG) attack against object detection transformers. By design, AFOG is neural-architecture agnostic and effective for attacking both large transformer-based object detectors and conventional CNN-based detectors with a unified adversarial attention framework. This paper makes three original contributions. First, AFOG utilizes a learnable attention mechanism that focuses perturbations on vulnerable image regions in multi-box detection tasks, increasing performance over non-attention baselines by up to 30.6%. Second, AFOG's attack loss is formulated by integrating two types of feature loss through learnable attention updates with iterative injection of adversarial perturbations. Finally, AFOG is an efficient and stealthy adversarial perturbation method. It probes the weak spots of detection transformers by adding strategically generated and visually imperceptible perturbations which can cause well-trained object detection models to fail. Extensive experiments conducted with twelve large detection transformers on COCO demonstrate the efficacy of AFOG. Our empirical results also show that AFOG outperforms existing attacks on transformer-based and CNN-based object detectors by up to 83% with superior speed and imperceptibility. Code is available at https://github.com/zacharyyahn/AFOG.

Problem

• Exposing vulnerabilities in transformer-based object detectors
• Unified adversarial attack for CNN and transformer detectors
• Generating stealthy perturbations to disrupt detection models

Innovation

• AFOG uses learnable attention for targeted perturbations
• Integrates feature loss with iterative adversarial updates
• Efficient stealthy attacks on transformer and CNN detectors