🤖 AI Summary
This study investigates how pretraining model scale, pretraining dataset size, and training strategies affect out-of-distribution (OOD) generalization and confidence calibration. We conduct controlled experiments across 100 models on four OOD benchmarks—ImageNet-C, -R, -A, and -O—accumulating over 120,000 GPU hours. Results reveal that pretraining model selection alone substantially improves OOD accuracy and calibration (reducing Expected Calibration Error by up to 40%), outperforming most dedicated OOD algorithms. Contrary to the prevailing belief that larger models degrade calibration, we find that scaling both model capacity and pretraining data jointly enhances calibration. Moreover, overconfidence systematically diminishes with increased model and data scale. This work provides the first large-scale empirical evidence demonstrating a dual positive effect of pretraining configuration—model architecture and data scale—on OOD robustness, establishing critical empirical foundations for designing trustworthy vision models.
📝 Abstract
In the field of computer vision, fine-tuning pre-trained models has become a prevalent strategy for out-of-distribution (OOD) generalization tasks. Different from most prior work that has focused on advancing learning algorithms, we systematically examined how pre-trained model size, pre-training dataset size, and training strategies impact generalization and confidence calibration on downstream tasks. We evaluated 100 models across diverse pre-trained model sizes, five pre-training datasets, and five data augmentations through extensive experiments on four distribution shift datasets totaling over 120,000 GPU hours. Our results demonstrate the significant impact of pre-trained model selection, with optimal choices substantially improving OOD accuracy over algorithm improvement alone. Additionally, we find that larger models and bigger pre-training datasets not only enhance OOD performance but also improve calibration, helping to mitigate overconfidence, contrary to some prior studies that found modern deep networks to calibrate worse than classical shallow models. Our work underscores the overlooked importance of pre-trained model selection for out-of-distribution generalization and calibration.