The Universal Weight Subspace Hypothesis

2025-12-04

AI Summary

This work investigates whether deep neural networks trained on diverse tasks, initializations, and domains share a common low-dimensional parameter subspace. Method: We perform systematic low-dimensional subspace extraction and comparison across over 1,100 model weight matrices—spanning heterogeneous architectures, tasks, and domains—using per-modal spectral analysis and singular value decomposition (SVD). Contribution/Results: We provide the first empirical evidence that, despite substantial differences in training configurations, models consistently converge to highly overlapping low-dimensional weight subspaces. Remarkably, fewer than 1% of the original parameter dimensions—captured by the top principal directions—are sufficient to explain over 90% of the parameter variance. This universal subspace exhibits strong reusability, enabling data-efficient parameter structure discovery, scalable multi-task learning, model fusion, and compression—thereby significantly reducing training cost and computational resource requirements.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

We show that deep neural networks trained across diverse tasks exhibit remarkably similar low-dimensional parametric subspaces. We provide the first large-scale empirical evidence that demonstrates that neural networks systematically converge to shared spectral subspaces regardless of initialization, task, or domain. Through mode-wise spectral analysis of over 1100 models - including 500 Mistral-7B LoRAs, 500 Vision Transformers, and 50 LLaMA-8B models - we identify universal subspaces capturing majority variance in just a few principal directions. By applying spectral decomposition techniques to the weight matrices of various architectures trained on a wide range of tasks and datasets, we identify sparse, joint subspaces that are consistently exploited, within shared architectures across diverse tasks and datasets. Our findings offer new insights into the intrinsic organization of information within deep networks and raise important questions about the possibility of discovering these universal subspaces without the need for extensive data and computational resources. Furthermore, this inherent structure has significant implications for model reusability, multi-task learning, model merging, and the development of training and inference-efficient algorithms, potentially reducing the carbon footprint of large-scale neural models.

Problem

• Identifies universal low-dimensional subspaces in diverse neural networks
• Demonstrates consistent convergence to shared spectral subspaces across tasks
• Explores implications for model efficiency and multi-task learning

Innovation

• Identifies universal low-dimensional weight subspaces across diverse neural networks
• Uses spectral decomposition to analyze weight matrices of various architectures
• Reveals shared spectral subspaces enabling efficient model reuse and training