A Recovery Guarantee for Sparse Neural Networks

2025-09-24

AI Summary

This work addresses the exact recovery of the original sparse weights in two-layer sparse ReLU neural networks. Existing methods suffer from high memory overhead and lack theoretical guarantees for recovery. To overcome these limitations, we propose a low-memory (linear-space) iterative hard thresholding (IHT) algorithm that jointly enforces sparsity constraints and performs gradient-based updates, enabling provably exact weight recovery without requiring predefined structural priors. To the best of our knowledge, this is the first work to provide rigorous theoretical guarantees for sparse weight recovery in ReLU networks. Experiments on sparse MLP reconstruction, MNIST classification, and implicit neural representations demonstrate that our method matches or surpasses the performance of memory-intensive iterative magnitude pruning baselines—while reducing memory consumption by an order of magnitude.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We prove the first guarantees of sparse recovery for ReLU neural networks, where the sparse network weights constitute the signal to be recovered. Specifically, we study structural properties of the sparse network weights for two-layer, scalar-output networks under which a simple iterative hard thresholding algorithm recovers these weights exactly, using memory that grows linearly in the number of nonzero weights. We validate this theoretical result with simple experiments on recovery of sparse planted MLPs, MNIST classification, and implicit neural representations. Experimentally, we find performance that is competitive with, and often exceeds, a high-performing but memory-inefficient baseline based on iterative magnitude pruning.

Problem

• Proving sparse recovery guarantees for ReLU neural networks
• Developing memory-efficient algorithm for recovering sparse weights
• Validating recovery performance against memory-inefficient pruning baseline

Innovation

• Proves sparse recovery guarantees for ReLU networks
• Uses iterative hard thresholding algorithm for recovery
• Achieves linear memory growth with nonzero weights