Training-Free Restoration of Pruned Neural Networks

2025-02-06

AI Summary

Pruning neural networks often leads to significant degradation in inference accuracy when fine-tuning or access to original training data is unavailable—especially when pruned neurons exhibit low similarity to retained ones, limiting the effectiveness of existing similarity-based recovery methods. To address this, we propose LBYL (Look Before You Leap), a novel reconstruction framework that abandons the strong “neuron substitutability” assumption underlying prior work. LBYL establishes the first theoretical model of reconstruction error and derives its closed-form solution. By dispersing information from pruned neurons across multiple retained neurons, it enables robust output reconstruction without requiring any retraining or original data—making it truly training-free and data-free. Extensive experiments across diverse architectures (e.g., ResNet, VGG) and datasets (e.g., ImageNet, CIFAR-10/100) demonstrate that LBYL consistently outperforms state-of-the-art neuron-similarity-based methods, achieving higher recovery accuracy with zero retraining overhead.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Although network pruning has been highly popularized to compress deep neural networks, its resulting accuracy heavily depends on a fine-tuning process that is often computationally expensive and requires the original data. However, this may not be the case in real-world scenarios, and hence a few recent works attempt to restore pruned networks without any expensive retraining process. Their strong assumption is that every neuron being pruned can be replaced with another one quite similar to it, but unfortunately this does not hold in many neural networks, where the similarity between neurons is extremely low in some layers. In this article, we propose a more rigorous and robust method of restoring pruned networks in a fine-tuning free and data-free manner, called LBYL (Leave Before You Leave). LBYL significantly relaxes the aforementioned assumption in a way that each pruned neuron leaves its pieces of information to as many preserved neurons as possible and thereby multiple neurons together obtain a more robust approximation to the original output of the neuron who just left. Our method is based on a theoretical analysis on how to formulate the reconstruction error between the original network and its approximation, which nicely leads to a closed form solution for our derived loss function. Through the extensive experiments, LBYL is confirmed to be indeed more effective to approximate the original network and consequently able to achieve higher accuracy for restored networks, compared to the recent approaches exploiting the similarity between two neurons. The very first version of this work, which contains major technical and theoretical components, was submitted to NeurIPS 2021 and ICML 2022.

Problem

• Restores pruned networks without retraining.
• Relaxes neuron similarity assumption for restoration.
• Enhances accuracy in data-free network restoration.

Innovation

• Training-free network restoration
• Data-free fine-tuning method
• Closed-form solution for loss function