A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy

2025-08-12

AI Summary

Systematic understanding of deep learning models’ robust generalization under out-of-distribution (OOD) perturbations remains limited. Method: We establish the most comprehensive empirical benchmark for robust fine-tuning to date, encompassing six datasets, 40 pre-trained architectures (including CNNs and Vision Transformers), multiple loss functions (e.g., adversarial and stable training), and parameter update protocols (full vs. partial fine-tuning), yielding 1,440 training configurations and 7,200 robustness evaluations. Contribution/Results: Contrary to the prevailing belief that Vision Transformers (ViTs) are inherently more robust, our results demonstrate that large-scale supervised pre-trained convolutional networks consistently outperform mainstream attention-based architectures across five OOD perturbation types. We uncover critical synergies among architecture design, pre-training methodology, and optimization strategy. The benchmark is fully reproducible and provides actionable, empirically grounded guidelines for improving model robustness—enabling practitioners to systematically enhance OOD generalization through informed architectural and training choices.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deep learning models operating in the image domain are vulnerable to small input perturbations. For years, robustness to such perturbations was pursued by training models from scratch (i.e., with random initializations) using specialized loss objectives. Recently, robust fine-tuning has emerged as a more efficient alternative: instead of training from scratch, pretrained models are adapted to maximize predictive performance and robustness. To conduct robust fine-tuning, practitioners design an optimization strategy that includes the model update protocol (e.g., full or partial) and the specialized loss objective. Additional design choices include the architecture type and size, and the pretrained representation. These design choices affect robust generalization, which is the model's ability to maintain performance when exposed to new and unseen perturbations at test time. Understanding how these design choices influence generalization remains an open question with significant practical implications. In response, we present an empirical study spanning 6 datasets, 40 pretrained architectures, 2 specialized losses, and 3 adaptation protocols, yielding 1,440 training configurations and 7,200 robustness measurements across five perturbation types. To our knowledge, this is the most diverse and comprehensive benchmark of robust fine-tuning to date. While attention-based architectures and robust pretrained representations are increasingly popular, we find that convolutional neural networks pretrained in a supervised manner on large datasets often perform best. Our analysis both confirms and challenges prior design assumptions, highlighting promising research directions and offering practical guidance.

Problem

• Understanding how design choices affect robust generalization to input perturbations
• Evaluating architecture, pre-training, and optimization strategies for robust fine-tuning
• Assessing model performance maintenance against unseen perturbations at test time

Innovation

• Robust fine-tuning with pretrained models
• Empirical study across diverse configurations
• Convolutional networks outperform attention-based architectures