A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints

2024-10-08

AI Summary

Pre-trained checkpoint selection lacks theoretical grounding, often relying on heuristic or downstream-dependent criteria. Method: This paper introduces “downstream free energy”—a task- and data-agnostic checkpoint adaptability metric grounded in Bayesian model selection. It leverages free-energy approximation and parameter-space concentration analysis to evaluate and rank checkpoints without accessing any downstream data or supervision. Contribution/Results: We formally define downstream free energy as a Bayesian adaptability measure, establishing the first meta-evaluation framework for pre-trained models that requires no downstream resources. Empirical evaluation across BERT, T5, and ViT on diverse tasks demonstrates strong correlation between downstream free energy and fine-tuning performance, significantly improving checkpoint selection accuracy. The framework provides an interpretable, generalizable, and theoretically principled foundation for pre-trained model selection.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent advances in artificial intelligence have been fueled by the development of foundation models such as BERT, GPT, T5, and Vision Transformers. These models are first pretrained on vast and diverse datasets and then adapted to specific downstream tasks, often with significantly less data. However, the mechanisms behind the success of this ubiquitous pretrain-then-adapt paradigm remain underexplored, particularly the characteristics of pretraining checkpoints that enhance downstream adaptation. We introduce a Bayesian model selection criterion, called the downstream free energy, which quantifies a checkpoint's adaptability by measuring the concentration of nearby favorable parameters for the downstream task. We demonstrate that this Bayesian model selection criterion can be effectively implemented without access to the downstream data or prior knowledge of the downstream task. Furthermore, we provide empirical evidence that the criterion reliably correlates with improved finetuning performance, offering a principled approach to predicting model adaptability.

Problem

• Identifying key pretraining checkpoint features for downstream tasks
• Quantifying checkpoint adaptability without downstream data access
• Predicting model adaptability via Bayesian selection criterion

Innovation

• Bayesian model selection for checkpoint adaptability
• Downstream free energy quantifies parameter favorability
• No downstream data needed for implementation