🤖 AI Summary
This work addresses the challenge of accurately estimating performance differences and their associated uncertainty when comparing randomly trained models, a task for which conventional methods are often inefficient. The authors propose a novel approach that leverages training logs as covariates to design model-specific covariate adjustment strategies. This method substantially reduces estimation uncertainty while preserving the original mean performance difference by performing posterior adjustment that effectively incorporates dynamic information from the training process. Crucially, it avoids introducing additional noise through careful covariate selection. Empirical evaluation across three model architectures and three datasets demonstrates that judicious use of early-stage training logs significantly enhances the statistical reliability of performance comparisons.
📝 Abstract
Comparing stochastically trained models requires estimating both a performance difference and its uncertainty from repeated runs. We study whether training logs from those same runs can make such comparisons more precise. Because training-log covariates are produced during training rather than measured before it, we use arm-specific covariate adjustment: each model is adjusted only with statistics from its own runs, and the raw mean difference remains the reported effect. In a vision study spanning three architectures and three datasets, simple adjustments based on early training logs often reduce uncertainty in model comparisons. The main limitation is covariate selection. Broadly searching the log pool for the most correlated statistic often adds more noise than it removes, even when useful statistics exist in hindsight. Training logs therefore appear useful for more precise model comparisons, but only when the adjustment avoids large selection noise.