On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networks

2024-12-09

arXiv.org

AI Summary

This study investigates how iterative magnitude pruning (IMP) spontaneously induces local receptive fields in fully connected networks (FCNs), addressing the puzzle of how IMP generates convolution-like structures in FCNs lacking explicit spatial priors. Method: We propose and validate a core mechanism—IMP fosters strong spatial inductive bias by feedback-enhancing non-Gaussianity in hidden-layer representations. We introduce the novel “cavity analysis” framework to quantify the causal influence of individual weights on representation statistics, integrated with non-Gaussianity measures, synthetic image training, and receptive field visualization. Contribution/Results: Empirical results demonstrate that IMP indeed constructs local receptive fields via statistical restructuring of representations. This work provides the first interpretable mechanistic account of inductive bias formation underlying the lottery ticket hypothesis, revealing a deep coupling between structural pruning and representation learning.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Since its use in the Lottery Ticket Hypothesis, iterative magnitude pruning (IMP) has become a popular method for extracting sparse subnetworks that can be trained to high performance. Despite this, the underlying nature of IMP's general success remains unclear. One possibility is that IMP is especially capable of extracting and maintaining strong inductive biases. In support of this, recent work has shown that applying IMP to fully connected neural networks (FCNs) leads to the emergence of local receptive fields (RFs), an architectural feature present in mammalian visual cortex and convolutional neural networks. The question of how IMP is able to do this remains unanswered. Inspired by results showing that training FCNs on synthetic images with highly non-Gaussian statistics (e.g., sharp edges) is sufficient to drive the formation of local RFs, we hypothesize that IMP iteratively increases the non-Gaussian statistics present in the representations of FCNs, creating a feedback loop that enhances localization. We develop a new method for measuring the effect of individual weights on the statistics of the FCN representations ("cavity method"), which allows us to find evidence in support of this hypothesis. Our work, which is the first to study the effect IMP has on the statistics of the representations of neural networks, sheds parsimonious light on one way in which IMP can drive the formation of strong inductive biases.

Problem

• IMP discovers local receptive fields
• IMP enhances neural network localization
• IMP increases non-Gaussian statistics in FCNs

Innovation

• Iterative Magnitude Pruning enhances localization
• Cavity method measures weight effects
• IMP increases non-Gaussian statistics in FCNs