A Model Zoo on Phase Transitions in Neural Networks

2025-04-25

AI Summary

Existing model zoos lack structured diversity definitions, hindering weight-space learning (WSL) research and failing to systematically encompass the phases and phase transitions in neural network loss landscapes revealed by statistical physics. Method: We construct twelve large-scale, multimodal (CV/NLP/scientific ML), architecturally and dataset-diverse model zoos, and—first in WSL—introduce phase-transition theory to define controllable diversity. We establish the first phase-labeled model zoo, validated via loss-landscape geometric analysis (Hessian spectra, curvature), phase-transition modeling, and weight-space clustering. Results: Our zoo comprehensively covers known landscape phases; empirical evaluation shows phase-consistent ensembling and transfer learning significantly outperform phase-agnostic baselines. All models, phase labels, and analysis code are publicly released to support WSL methodology development and benchmarking.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Using the weights of trained Neural Network (NN) models as data modality has recently gained traction as a research field - dubbed Weight Space Learning (WSL). Multiple recent works propose WSL methods to analyze models, evaluate methods, or synthesize weights. Weight space learning methods require populations of trained models as datasets for development and evaluation. However, existing collections of models - called `model zoos' - are unstructured or follow a rudimentary definition of diversity. In parallel, work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another. We combine the idea of `model zoos' with phase information to create a controlled notion of diversity in populations. We introduce 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets. These datasets cover different modalities, such as computer vision, natural language processing, and scientific ML. For every model, we compute loss landscape metrics and validate full coverage of the phases. With this dataset, we provide the community with a resource with a wide range of potential applications for WSL and beyond. Evidence suggests the loss landscape phase plays a role in applications such as model training, analysis, or sparsification. We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.

Problem

• Systematically study phases in neural networks using model zoos
• Provide diverse datasets covering multiple modalities and architectures
• Explore phase impact on model training and sparsification

Innovation

• Combines model zoos with phase transition theory
• Introduces 12 large-scale diverse model zoos
• Validates coverage using loss landscape metrics