Robustness Tokens: Towards Adversarial Robustness of Transformers

2025-03-13

European Conference on Computer Vision

AI Summary

This work addresses the vulnerability of Transformer backbone models to adversarial attacks. We propose Learnable Robustness Tokens—a plug-and-play, lightweight defense module that requires no architectural modification or full-model retraining. Our method introduces trainable attention-guided token embeddings, optimized end-to-end via joint adversarial training, gradient masking, and feature disentanglement. Crucially, it is the first to employ learnable tokens as a general-purpose robustness enhancement mechanism, dynamically injecting salient defensive features into the representation space. Evaluated on ImageNet-C and AutoAttack benchmarks, our approach achieves an average robust accuracy improvement of 12.3%, with less than 0.5% degradation in clean accuracy—substantially outperforming existing plug-and-play defenses.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Problem

• Enhance adversarial robustness of transformer models
• Reduce vulnerability to white-box adversarial attacks
• Maintain original performance with low computational cost

Innovation

• Introduces Robustness Tokens for transformer models
• Fine-tunes private tokens, not model parameters
• Enhances robustness against white-box adversarial attacks