Feature Space Perturbation: A Panacea to Enhanced Transferability Estimation

2025-02-23

AI Summary

Current transferability evaluation methods for pretrained models often neglect robustness, leading to inaccurate downstream task rankings. To address this, this paper proposes a novel transferability assessment framework that explicitly incorporates robustness modeling. Its core innovation is the “Spread-Attract” dual-operation collaborative perturbation mechanism: it systematically perturbs representations in feature space to simultaneously maximize intra-class dispersion (Spread) and minimize inter-class distance (Attract), thereby yielding a more reliable characterization of a model’s generalization capability on target domains. Integrated into the LogMe transferability metric, this mechanism significantly improves predictive accuracy—achieving a 28.84% gain in transferability prediction performance on standard benchmarks. Moreover, it substantially enhances both cross-model ranking accuracy and robustness, demonstrating superior consistency and reliability in practical transfer scenarios.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Leveraging a transferability estimation metric facilitates the non-trivial challenge of selecting the optimal model for the downstream task from a pool of pre-trained models. Most existing metrics primarily focus on identifying the statistical relationship between feature embeddings and the corresponding labels within the target dataset, but overlook crucial aspect of model robustness. This oversight may limit their effectiveness in accurately ranking pre-trained models. To address this limitation, we introduce a feature perturbation method that enhances the transferability estimation process by systematically altering the feature space. Our method includes a Spread operation that increases intra-class variability, adding complexity within classes, and an Attract operation that minimizes the distances between different classes, thereby blurring the class boundaries. Through extensive experimentation, we demonstrate the efficacy of our feature perturbation method in providing a more precise and robust estimation of model transferability. Notably, the existing LogMe method exhibited a significant improvement, showing a 28.84% increase in performance after applying our feature perturbation method.

Problem

• Enhance transferability estimation accuracy
• Improve model robustness evaluation
• Optimize pre-trained model selection

Innovation

• Feature perturbation enhances transferability estimation
• Spread operation increases intra-class variability
• Attract operation minimizes inter-class distances