CIARD: Cyclic Iterative Adversarial Robustness Distillation

2025-09-15

AI Summary

Existing adversarial robustness distillation (ARD) methods often suffer from teacher performance degradation due to conflicting dual-teacher objectives and iterative adversarial sample generation, thereby compromising clean-accuracy. This paper proposes a multi-teacher collaborative distillation framework. First, we introduce a contrastive push-away loss to mitigate optimization objective conflicts among teachers. Second, we design a cyclic adversarial retraining mechanism to dynamically preserve teacher robustness. By integrating multi-teacher ensembling, adversarial distillation, and contrastive learning, our approach avoids the performance decay inherent in conventional iterative adversarial sample generation. Extensive experiments on CIFAR-10/100 and Tiny-ImageNet demonstrate that our method achieves an average 3.53% improvement in adversarial accuracy and a 5.87% gain in clean accuracy over state-of-the-art methods.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Adversarial robustness distillation (ARD) aims to transfer both performance and robustness from teacher model to lightweight student model, enabling resilient performance on resource-constrained scenarios. Though existing ARD approaches enhance student model's robustness, the inevitable by-product leads to the degraded performance on clean examples. We summarize the causes of this problem inherent in existing methods with dual-teacher framework as: 1. The divergent optimization objectives of dual-teacher models, i.e., the clean and robust teachers, impede effective knowledge transfer to the student model, and 2. The iteratively generated adversarial examples during training lead to performance deterioration of the robust teacher model. To address these challenges, we propose a novel Cyclic Iterative ARD (CIARD) method with two key innovations: a. A multi-teacher framework with contrastive push-loss alignment to resolve conflicts in dual-teacher optimization objectives, and b. Continuous adversarial retraining to maintain dynamic teacher robustness against performance degradation from the varying adversarial examples. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CIARD achieves remarkable performance with an average 3.53 improvement in adversarial defense rates across various attack scenarios and a 5.87 increase in clean sample accuracy, establishing a new benchmark for balancing model robustness and generalization. Our code is available at https://github.com/eminentgu/CIARD

Problem

• Resolve dual-teacher optimization conflicts in robustness distillation
• Mitigate performance degradation from iterative adversarial examples
• Balance model robustness and generalization in lightweight students

Innovation

• Multi-teacher framework with contrastive push-loss alignment
• Continuous adversarial retraining for dynamic robustness
• Cyclic iterative distillation balancing robustness and accuracy