Squeeze-Release: Iterative Pruning with Exact Structural Minimization

2026-06-12

AI Summary

This work addresses the limitation of unstructured pruning, which yields sparse weights but fails to reduce model size in deployment due to preserved tensor shapes. The authors propose a Squeeze-Release cycle: first, an exact structured minimization compresses the masked network into a smaller dense one; then, a parameter release step reactivates zero-valued positions to iteratively uncover deeper structural redundancy. Additionally, they introduce CompensatedLayerNorm, the first method enabling channel compression across LayerNorm residual streams. The approach generalizes to Transformer architectures, achieving 39× and 14.8× deployment-size compression on fully connected networks and ConvNeXt-Tiny, respectively, with negligible accuracy loss.

Abstract

Unstructured pruning produces sparse weight tensors, but the standard implementation keeps tensor shapes unchanged so the deployed model is no smaller than before pruning. We present an exact structural rewrite, which we call minimization, that converts a masked network into a smaller dense network with the same forward function up to floating-point rounding. The Squeeze-Release cycle iterates pruning and minimization with an intermediate release step that re-enables the exact-zero positions inside the compacted tensors as small calibrated noise, turning otherwise wasted capacity back into trainable parameters. Successive cycles use that capacity to find structural redundancy a single pass cannot reach. We additionally introduce CompensatedLayerNorm, a function-preserving replacement for LayerNorm that extends minimization to channel reduction across LayerNorm-equipped residual streams. Squeeze-Release compresses the deployable network to 39x smaller than the unpruned model on a fully-connected model network and 14.8x smaller on modern CNN (ConvNeXt-Tiny), at comparable accuracy. In addition we prove that the rewrite can be extended to transformer architectures.

Problem

• unstructured pruning
• model compression
• structural minimization
• dense network conversion
• LayerNorm

Innovation

• structural minimization
• iterative pruning
• Squeeze-Release
• CompensatedLayerNorm
• model compression