PAODING: A High-fidelity Data-free Pruning Toolkit for Debloating Pre-trained Neural Networks

2024-04-30

arXiv.org

AI Summary

To address the challenge of high redundancy in pretrained neural networks coupled with unavailability of original training data, this paper proposes PAODING—a data-free, high-fidelity pruning method. PAODING dynamically quantifies each neuron’s gradient-based influence on the output layer and iteratively identifies and removes the least-contributing structured units via sensitivity analysis. It introduces the first unsupervised neuron importance scoring mechanism, requiring neither training data nor labels, while simultaneously preserving model accuracy and adversarial robustness. Extensive evaluation across diverse architectures—including ViT and ResNet—demonstrates that PAODING achieves significant model compression, incurs less than 0.5% accuracy drop on standard test sets, maintains near-identical adversarial robustness, and supports cross-model and cross-dataset generalizable pruning.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We present PAODING, a toolkit to debloat pretrained neural network models through the lens of data-free pruning. To preserve the model fidelity, PAODING adopts an iterative process, which dynamically measures the effect of deleting a neuron to identify candidates that have the least impact to the output layer. Our evaluation shows that PAODING can significantly reduce the model size, generalize on different datasets and models, and meanwhile preserve the model fidelity in terms of test accuracy and adversarial robustness. PAODING is publicly available on PyPI via https://pypi.org/project/paoding-dl.

Problem

• Debloating pretrained neural networks without data
• Preserving model fidelity during pruning
• Reducing model size while maintaining accuracy

Innovation

• Data-free pruning for neural networks
• Iterative neuron impact measurement
• Preserves model fidelity and robustness