Boosting Adversarial Transferability via Ensemble Non-Attention

2025-11-12

AI Summary

To address the low transferability of adversarial examples across heterogeneous architectures, this paper proposes NAMEA—a novel ensemble-based black-box attack method. NAMEA is the first to empirically identify and exploit the complementary contribution of non-attention regions in cross-architecture adversarial transfer. Leveraging the structural complementarity between Vision Transformer (ViT) and Convolutional Neural Network (CNN) attention mechanisms, it introduces a meta-learning-driven gradient decoupling and fusion framework that separately models and jointly optimizes adversarial gradients for attention and non-attention regions. This design effectively mitigates gradient direction conflicts among heterogeneous models, thereby enhancing transfer robustness. Evaluated on ImageNet, NAMEA achieves average attack success rates 15.0% higher than AdaEA and 9.6% higher than SMER—outperforming all existing state-of-the-art ensemble attacks. The method establishes a new paradigm for cross-architecture black-box adversarial attacks.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Ensemble attacks integrate the outputs of surrogate models with diverse architectures, which can be combined with various gradient-based attacks to improve adversarial transferability. However, previous work shows unsatisfactory attack performance when transferring across heterogeneous model architectures. The main reason is that the gradient update directions of heterogeneous surrogate models differ widely, making it hard to reduce the gradient variance of ensemble models while making the best of individual model. To tackle this challenge, we design a novel ensemble attack, NAMEA, which for the first time integrates the gradients from the non-attention areas of ensemble models into the iterative gradient optimization process. Our design is inspired by the observation that the attention areas of heterogeneous models vary sharply, thus the non-attention areas of ViTs are likely to be the focus of CNNs and vice versa. Therefore, we merge the gradients respectively from the attention and non-attention areas of ensemble models so as to fuse the transfer information of CNNs and ViTs. Specifically, we pioneer a new way of decoupling the gradients of non-attention areas from those of attention areas, while merging gradients by meta-learning. Empirical evaluations on ImageNet dataset indicate that NAMEA outperforms AdaEA and SMER, the state-of-the-art ensemble attacks by an average of 15.0% and 9.6%, respectively. This work is the first attempt to explore the power of ensemble non-attention in boosting cross-architecture transferability, providing new insights into launching ensemble attacks.

Problem

• Improving adversarial transferability across heterogeneous model architectures
• Reducing gradient variance in ensemble models through non-attention areas
• Merging attention and non-attention gradients via meta-learning optimization

Innovation

• Integrates gradients from non-attention areas of models
• Merges attention and non-attention gradients via meta-learning
• Decouples gradients to fuse CNN and ViT transfer information