A Survey of Weight Space Learning: Understanding, Representation, and Generation

2026-03-10

AI Summary

This work addresses the long-standing neglect of the structural and knowledge-rich properties inherent in neural network weights, as well as the lack of systematic investigation into weight space within conventional deep learning. To bridge this gap, the paper introduces Weight Space Learning (WSL), a unified framework that establishes the first comprehensive taxonomy for studying weight space through three core dimensions: understanding (geometric structure and symmetries), representation (model embeddings), and generation (hypernetworks and generative models). By integrating previously fragmented research efforts, WSL reveals the potential of weights as a learnable, structured domain and enables advances in diverse applications—including model retrieval, continual learning, federated learning, neural architecture search, and data-free reconstruction. The authors further support community progress by releasing an open-source repository dedicated to weight space research.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

Problem

• Weight Space Learning
• Neural Network Weights
• Weight Space Structure
• Model Analysis
• Knowledge Transfer

Innovation

• Weight Space Learning
• Weight Space Representation
• Weight Space Generation
• Model Embedding
• Hypernetworks