Denoised Variance-Based Pruning with Optimal Brain Bias Compensation

2026-08-18

AI Summary

为解决视觉转换器在边缘部署时的高计算成本问题，提出了一种基于去噪方差的剪枝方法DVBP + OB²C，通过去除激活协方差谱中的噪声并优化权重更新来减少精度损失。

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance but carry massive computational overhead that restricts edge deployment. Although structural pruning has emerged as a key strategy to reduce these costs, existing methods often suffer from severe accuracy degradation or require expensive retraining. Recently, Variance-Based Pruning (VBP) introduced a promising paradigm by selecting neurons based on activation variance; however, it remains limited by statistical noise in finite-sample activation covariance and reliance on bias-only updates that cannot fully account for structural reconstruction error. To address these limitations, we introduce Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$^2$C). We leverage random matrix theory to filter noise from the activation covariance spectrum for robust neuron selection and mathematically prove that integrating mean-shift compensation into the Optimal Brain Compression objective reduces the layer-wise Hessian exactly to the activation covariance matrix. This enables an optimal, closed-form update of the remaining weights using the same statistics gathered for selection. Extensive experiments on DeiT, Swin, and ConvNeXt architectures demonstrate that DVBP + OB$^2$C achieves state-of-the-art training-free performance; at 50% MLP pruning, it retains over 90% of the original Top-1 accuracy on Small and Base variants, outperforming VBP by up to 29.46% (ConvNeXt-T) and 7.33% (Swin-S). The code is available at: https://github.com/geontackee/DVBP_OB2C.

Problem

• Vision Transformers
• computational overhead
• pruning
• accuracy degradation
• retraining

Innovation

• Denoised Variance-Based Pruning
• Optimal Brain Bias Compensation
• Random Matrix Theory
• Activation Covariance
• Training-free Performance