Benign Overfitting in Adversarial Training for Vision Transformers

2026-04-21

AI Summary

Vision Transformers (ViTs) currently lack theoretically grounded robustness guarantees under adversarial attacks. This work addresses this gap by analyzing a simplified ViT architecture and, for the first time, theoretically establishes that adversarial training can induce benign overfitting in ViTs—extending this phenomenon from convolutional neural networks to the Transformer domain. Through a combination of theoretical analysis and empirical validation on both synthetic and real-world datasets, we demonstrate that under specific signal-to-noise ratios and perturbation budgets, the model achieves near-zero robust training loss alongside low robust generalization error, thereby confirming the validity of our theoretical predictions.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Despite the remarkable success of Vision Transformers (ViTs) across a wide range of vision tasks, recent studies have revealed that they remain vulnerable to adversarial examples, much like Convolutional Neural Networks (CNNs). A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored. In this work, we present the first theoretical analysis of adversarial training under simplified ViT architectures. We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate perturbation budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes. Remarkably, this leads to strong generalization even in the presence of overfitting, a phenomenon known as \emph{benign overfitting}, previously only observed in CNNs (with adversarial training). Experiments on both synthetic and real-world datasets further validate our theoretical findings.

Problem

• Vision Transformers
• adversarial training
• benign overfitting
• robustness
• adversarial examples

Innovation

• benign overfitting
• adversarial training
• Vision Transformers
• robust generalization
• theoretical analysis