Learning Model Representations Using Publicly Available Model Hubs

2025-10-02

AI Summary

This work addresses the critical bottleneck in weight-space representation learning—its heavy reliance on manually curated, computationally expensive “model zoos.” We propose the first weight representation learning framework designed for open model repositories (e.g., Hugging Face). Methodologically, we introduce an architecture-agnostic weight alignment and normalization mechanism, integrated with a self-supervised backbone network, to learn robust weight embeddings directly from large-scale, heterogeneous, unlabeled, cross-architecture, and cross-task public models. Our key contributions are: (1) the first successful learning of high-quality weight representations in the “wild-model” setting—eliminating dependence on curated model zoos entirely; and (2) state-of-the-art or superior performance on downstream tasks—including model retrieval, performance prediction, and weight interpolation—while demonstrating strong cross-modal generalization, outperforming prior methods reliant on controlled laboratory model collections.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The weights of neural networks have emerged as a novel data modality, giving rise to the field of weight space learning. A central challenge in this area is that learning meaningful representations of weights typically requires large, carefully constructed collections of trained models, typically referred to as model zoos. These model zoos are often trained ad-hoc, requiring large computational resources, constraining the learned weight space representations in scale and flexibility. In this work, we drop this requirement by training a weight space learning backbone on arbitrary models downloaded from large, unstructured model repositories such as Hugging Face. Unlike curated model zoos, these repositories contain highly heterogeneous models: they vary in architecture and dataset, and are largely undocumented. To address the methodological challenges posed by such heterogeneity, we propose a new weight space backbone designed to handle unstructured model populations. We demonstrate that weight space representations trained on models from Hugging Face achieve strong performance, often outperforming backbones trained on laboratory-generated model zoos. Finally, we show that the diversity of the model weights in our training set allows our weight space model to generalize to unseen data modalities. By demonstrating that high-quality weight space representations can be learned in the wild, we show that curated model zoos are not indispensable, thereby overcoming a strong limitation currently faced by the weight space learning community.

Problem

• Learning weight representations from heterogeneous public model repositories
• Overcoming limitations of curated model zoos requiring massive computational resources
• Developing methods to handle unstructured models with varying architectures and datasets

Innovation

• Leveraging unstructured models from public repositories
• Designing backbone for heterogeneous model populations
• Achieving generalization across unseen data modalities