🤖 AI Summary
Deep neural network training suffers from high sensitivity to random seeds due to stochastic optimization, hindering reliable assessment of true generalization performance. To address this, we propose a robust nonparametric hypothesis testing framework. Its core innovation is a novel model similarity metric—the α-truncation level—which quantifies training variability and determines the minimum number of independent training runs required for stable ensembling. Unlike conventional metrics such as accuracy or expected calibration error (ECE), the α-truncation level does not rely on modeling the null distribution and is inherently sensitive to training instability. Experiments demonstrate that it detects training uncertainty earlier and more consistently than validation accuracy, churn, and ECE. Moreover, in transfer learning settings, it effectively guides random seed selection, significantly improving the reliability of performance evaluation.
📝 Abstract
Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the $alpha$-trimming level, to measure model similarity. Applying hypothesis testing directly with the $alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the $alpha$-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the $alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.