Free Decompression with Algebraic Spectral Curves

2026-05-05

AI Summary

Existing free deconvolution methods struggle to capture the complex spectral structures of real-world machine learning models due to restrictive assumptions and scalability limitations. This work reframes free deconvolution as an integrable spectral curve evolution problem, grounded in algebraic spectral curve theory, wherein the Stieltjes transform satisfies an algebraic relation. The resulting general framework accommodates intricate spectral features—including multimodality, multiscale behavior, and atomic components—thereby overcoming traditional constraints. The approach is successfully applied to Hessian and activation matrices of neural networks as well as large-scale diffusion models, enabling effective cross-scale spectral extrapolation and significantly enhancing both the modeling fidelity and practical utility for characterizing the spectral properties of realistic models.

Abstract

Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and failure modes. While often capable of modeling empirical behavior, practical computations are limited by matrix size, often imposing a restriction to models that are too small to be realistic. This motivates the inference of properties of larger models from the behavior of smaller ones. Free decompression (FD) is a recently proposed method for extrapolating spectral information across matrix sizes, but its utility is currently limited by strong assumptions that preclude its implementation on more realistic machine learning (ML) models. We use algebraic spectral curve theory to provide a general FD methodology for spectral densities whose Stieltjes transform satisfies an algebraic relation, a modeling assumption that is more likely to hold in practice. This recasts FD as an evolution along spectral curves which can be readily integrated. Our framework enables the expansion of spectral densities that have multiple or multi-modal bulks, that exist at multiple scales, and that contain atoms, all characteristic of real-world data and popular ML models. We demonstrate the efficacy of our framework on models of interest in modern ML, including Hessian and activation matrices associated with neural networks and large-scale diffusion models.

Problem

• free decompression
• spectral curves
• random matrix theory
• spectral density
• machine learning models

Innovation

• Free Decompression
• Algebraic Spectral Curves
• Stieltjes Transform
• Spectral Density
• Random Matrix Theory