Width-Independent Compressibility of Deep Neural Networks

2026-08-21

AI Summary

研究证明了深度多层感知器的均匀可压缩性，通过一种新的导数匹配技术和逐层重加权方法，使得网络宽度可以独立于原始宽度而大幅减小。

Abstract

It has long been known that well-trained neural networks can be compressed very strongly without affecting their performance, an important phenomenon that remains poorly understood. We prove a uniform compressibility theorem for deep multilayer perceptrons with analytic activations. For a deep, wide fixed teacher network, there exists a narrow (same depth) network that approximately represents the same function as the original. The reachable compressed width is strikingly independent of the original width, but is $O((\log(1/\varepsilon))^{d_{in}})$, where $\varepsilon$ is the error budget and $d_{in}$ is the effective input dimension. Our construction involves a novel derivative-matching technique which is aware of the low-dimensional input, and a layer-wise reweighting that preserves the input-output mapping.

Problem

• deep neural networks
• compressibility
• multilayer perceptrons
• analytic activations

Innovation

• uniform compressibility theorem
• derivative-matching technique
• layer-wise reweighting