DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights

2026-01-08

arXiv.org

AI Summary

This work addresses the challenges of generative modeling in high-dimensional neural network weight spaces, where high dimensionality and strong permutation symmetries hinder existing methods from efficiently producing complete, high-performance large-scale network weights. The paper introduces DeepWeightFlow, which for the first time integrates Git Re-Basin with Flow Matching and incorporates TransFusion for weight normalization, effectively eliminating weight symmetries and enabling direct generation of diverse, high-accuracy network parameters in the original weight space. Without requiring fine-tuning, the method achieves strong transfer performance across diverse architectures and data modalities, generating hundreds of high-performing models within minutes. It substantially outperforms diffusion-based approaches, marking a significant advance in both generation efficiency and diversity.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Building efficient and effective generative models for neural network weights has been a research focus of significant interest that faces challenges posed by the high-dimensional weight spaces of modern neural networks and their symmetries. Several prior generative models are limited to generating partial neural network weights, particularly for larger models, such as ResNet and ViT. Those that do generate complete weights struggle with generation speed or require finetuning of the generated models. In this work, we present DeepWeightFlow, a Flow Matching model that operates directly in weight space to generate diverse and high-accuracy neural network weights for a variety of architectures, neural network sizes, and data modalities. The neural networks generated by DeepWeightFlow do not require fine-tuning to perform well and can scale to large networks. We apply Git Re-Basin and TransFusion for neural network canonicalization in the context of generative weight models to account for the impact of neural network permutation symmetries and to improve generation efficiency for larger model sizes. The generated networks excel at transfer learning, and ensembles of hundreds of neural networks can be generated in minutes, far exceeding the efficiency of diffusion-based methods. DeepWeightFlow models pave the way for more efficient and scalable generation of diverse sets of neural networks.

Problem

• neural network weight generation
• high-dimensional weight space
• permutation symmetries
• generative modeling
• weight space canonicalization

Innovation

• Flow Matching
• Weight Generation
• Git Re-Basin
• Permutation Symmetry
• Neural Network Canonicalization