Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling

2025-09-27

AI Summary

This paper addresses the suboptimal transfer problem in robust fine-tuning (RFT) of non-robust pre-trained models—specifically, the degradation of downstream task adaptability caused by overly strong adversarial objectives. To mitigate this, we propose Epsilon-Scheduling: a dynamic perturbation decay strategy that balances robustness and task-specific performance. We further introduce Expected Robustness, a novel metric that quantifies the accuracy–robustness trade-off across varying perturbation magnitudes. Extensive experiments across six mainstream pre-trained models and five benchmark datasets demonstrate that our approach significantly improves Expected Robustness (average gain of +12.7%) while effectively alleviating the robustness–adaptability trade-off. The method yields stable, efficient, and reproducible robust fine-tuning outcomes.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Fine-tuning pretrained models is a standard and effective workflow in modern machine learning. However, robust fine-tuning (RFT), which aims to simultaneously achieve adaptation to a downstream task and robustness to adversarial examples, remains challenging. Despite the abundance of non-robust pretrained models in open-source repositories, their potential for RFT is less understood. We address this knowledge gap by systematically examining RFT from such non-robust models. Our experiments reveal that fine-tuning non-robust models with a robust objective, even under small perturbations, can lead to poor performance, a phenomenon that we dub emph{suboptimal transfer}. In challenging scenarios (eg, difficult tasks, high perturbation), the resulting performance can be so low that it may be considered a transfer failure. We find that fine-tuning using a robust objective impedes task adaptation at the beginning of training and eventually prevents optimal transfer. However, we propose a novel heuristic, emph{Epsilon-Scheduling}, a schedule over perturbation strength used during training that promotes optimal transfer. Additionally, we introduce emph{expected robustness}, a metric that captures performance across a range of perturbations, providing a more comprehensive evaluation of the accuracy-robustness trade-off for diverse models at test time. Extensive experiments on a wide range of configurations (six pretrained models and five datasets) show that emph{Epsilon-Scheduling} successfully prevents emph{suboptimal transfer} and consistently improves expected robustness.

Problem

• Addressing suboptimal transfer in robust fine-tuning from non-robust pretrained models
• Mitigating poor performance when adapting non-robust models to adversarial robustness
• Improving expected robustness across diverse tasks and perturbation scenarios

Innovation

• Epsilon-Scheduling heuristic for perturbation strength
• Adversarial scheduling prevents suboptimal transfer
• Expected robustness metric evaluates accuracy-robustness trade-off