Towards a mathematical theory of superposition

2026-08-27

AI Summary

本文利用框架理论和压缩感知工具，建立了神经网络叠加的数学理论，并证明了在不同条件下特征恢复的可能性。

Abstract

We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector \(x\) of active features is encoded through an overcomplete dictionary \(W\), and feature recovery is performed by applying \(\operatorname{ReLU}(W^\top W x+b)\) with an appropriate bias vector \(b\). We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order \(d/\log n\). In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with \(n>d+1\), we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.

Problem

• superposition
• neural networks
• sparse representation
• feature recovery
• overcomplete dictionary

Innovation

• frame theory
• compressed sensing
• support recovery
• equiangular tight frames
• Gram matrix