Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

2025-06-06

AI Summary

This work addresses the bias in heavy-tailedness estimation arising from aspect-ratio disparities in weight matrices during spectral analysis of deep neural networks. We propose Fixed-Aspect-Ratio Matrix Sampling (FARMS), a method that mitigates this bias by randomly sampling submatrices with fixed aspect ratios, modeling their empirical spectral density (ESD), and fitting α-stable distributions to estimate the tail index. FARMS is the first framework to systematically eliminate the intrinsic aspect-ratio-induced bias in spectral statistics. It exhibits strong cross-architecture and cross-task robustness, significantly improving model diagnostics and layer-wise hyperparameter allocation. Extensive validation across computer vision (CV), scientific machine learning (SciML), and large language model (LLM) pruning tasks confirms its effectiveness: when applied to LLaMA-7B pruning, FARMS reduces perplexity by 17.3%, outperforming state-of-the-art methods.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Computer Vision: Other Foundations of Computer Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Diagnosing deep neural networks (DNNs) through the eigenspectrum of weight matrices has been an active area of research in recent years. At a high level, eigenspectrum analysis of DNNs involves measuring the heavytailness of the empirical spectral densities (ESD) of weight matrices. It provides insight into how well a model is trained and can guide decisions on assigning better layer-wise training hyperparameters. In this paper, we address a challenge associated with such eigenspectrum methods: the impact of the aspect ratio of weight matrices on estimated heavytailness metrics. We demonstrate that matrices of varying sizes (and aspect ratios) introduce a non-negligible bias in estimating heavytailness metrics, leading to inaccurate model diagnosis and layer-wise hyperparameter assignment. To overcome this challenge, we propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), a method that normalizes the weight matrices by subsampling submatrices with a fixed aspect ratio. Instead of measuring the heavytailness of the original ESD, we measure the average ESD of these subsampled submatrices. We show that measuring the heavytailness of these submatrices with the fixed aspect ratio can effectively mitigate the aspect ratio bias. We validate our approach across various optimization techniques and application domains that involve eigenspectrum analysis of weights, including image classification in computer vision (CV) models, scientific machine learning (SciML) model training, and large language model (LLM) pruning. Our results show that despite its simplicity, FARMS uniformly improves the accuracy of eigenspectrum analysis while enabling more effective layer-wise hyperparameter assignment in these application domains. In one of the LLM pruning experiments, FARMS reduces the perplexity of the LLaMA-7B model by 17.3% when compared with the state-of-the-art method.

Problem

• Addressing bias in eigenspectrum analysis of neural networks
• Mitigating aspect ratio impact on heavytailness metrics
• Improving layer-wise hyperparameter assignment accuracy

Innovation

• Subsamples matrices with fixed aspect ratio
• Averages ESD of subsampled submatrices
• Mitigates bias in eigenspectrum analysis