LEAD: Exploring Logit Space Evolution for Model Selection

2025-07-19

AI Summary

To address the challenges of inefficient pre-trained model selection and reliance on time-consuming fine-tuning for downstream tasks, this paper proposes a transferability prediction method based on nonlinear evolution modeling in the logits space. We establish, for the first time, an ordinary differential equation (ODE) theoretical framework for logits evolution from the perspective of optimization objective alignment, and introduce a class-aware decomposition mechanism to enable fine-grained, tuning-free transferability assessment. Our method directly leverages the initial logits produced by pre-trained models on downstream data to simulate the dynamic evolution trajectory during fine-tuning, thereby enabling efficient prediction of final task performance. Extensive experiments across 10 downstream datasets and 24 pre-trained models demonstrate that our approach significantly improves model selection efficiency, outperforming existing zero-shot and few-shot evaluation baselines—especially under low-resource conditions.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The remarkable success of pretrain-then-finetune paradigm has led to a proliferation of available pre-trained models for vision tasks. This surge presents a significant challenge in efficiently choosing the most suitable pre-trained models for downstream tasks. The critical aspect of this challenge lies in effectively predicting the model transferability by considering the underlying fine-tuning dynamics. Existing methods often model fine-tuning dynamics in feature space with linear transformations, which do not precisely align with the fine-tuning objective and fail to grasp the essential nonlinearity from optimization. To this end, we present LEAD, a finetuning-aligned approach based on the network output of logits. LEAD proposes a theoretical framework to model the optimization process and derives an ordinary differential equation (ODE) to depict the nonlinear evolution toward the final logit state. Additionally, we design a class-aware decomposition method to consider the varying evolution dynamics across classes and further ensure practical applicability. Integrating the closely aligned optimization objective and nonlinear modeling capabilities derived from the differential equation, our method offers a concise solution to effectively bridge the optimization gap in a single step, bypassing the lengthy fine-tuning process. The comprehensive experiments on 24 supervised and self-supervised pre-trained models across 10 downstream datasets demonstrate impressive performances and showcase its broad adaptability even in low-data scenarios.

Problem

• Efficiently selecting suitable pre-trained models for downstream tasks
• Predicting model transferability by analyzing fine-tuning dynamics
• Modeling nonlinear logit evolution to bridge optimization gap

Innovation

• Models logit space evolution via ODE
• Class-aware decomposition for dynamics
• Single-step optimization gap bridging