🤖 AI Summary
ℓ∞-adversarial training (ℓ∞-AT) induces imbalanced input gradient distributions, rendering models overly sensitive to perturbations on salient pixels and degrading their generalization robustness against occlusions, in-distribution noise, and natural corruptions (e.g., ImageNet-C). To address this, we propose Input Gradient Distillation (IGD), the first method that explicitly equalizes saliency map distributions via input gradient distillation—thereby theoretically mitigating gradient imbalance. IGD is plug-and-play, parameter-free, and seamlessly integrates with PGD-based adversarial training (PGDAT). Experiments demonstrate that, compared to standard ℓ∞-AT, IGD reduces error rates by 16.53% on occlusion benchmarks, 60% on in-distribution noise, and 21.11% on ImageNet-C, while fully preserving the original ℓ∞-adversarial accuracy. This yields substantial improvements in robust generalization without compromising worst-case adversarial robustness.
📝 Abstract
Since adversarial examples appeared and showed the catastrophic degradation they brought to DNN, many adversarial defense methods have been devised, among which adversarial training is considered the most effective. However, a recent work showed the inequality phenomena in $l_{infty}$-adversarial training and revealed that the $l_{infty}$-adversarially trained model is vulnerable when a few important pixels are perturbed by i.i.d. noise or occluded. In this paper, we propose a simple yet effective method called Input Gradient Distillation (IGD) to release the inequality phenomena in $l_{infty}$-adversarial training. Experiments show that while preserving the model's adversarial robustness, compared to PGDAT, IGD decreases the $l_{infty}$-adversarially trained model's error rate to inductive noise and inductive occlusion by up to 60% and 16.53%, and to noisy images in Imagenet-C by up to 21.11%. Moreover, we formally explain why the equality of the model's saliency map can improve such robustness.
