Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks

2026-06-26

AI Summary

This work addresses the severe accuracy degradation in deep binary neural networks (BNNs) trained from scratch, which stems from the inconsistency between forward and backward passes inherent in the straight-through estimator (STE). The authors propose StoMPP, a novel training framework that forgoes STE entirely and reveals, for the first time, the critical role of binarization order: progressive layer-wise binarization in the forward pass prevents depth-induced collapse, whereas reverse ordering leads to catastrophic performance failure. StoMPP integrates stochastic partial masking, a soft-refresh mechanism, and inter-layer progressive freezing to enable stable end-to-end training. On ResNet-50, it achieves accuracy gains of 18.0%, 13.5%, and 3.8% on CIFAR-10, CIFAR-100, and ImageNet, respectively; when combined with surrogate gradients, improvements reach 27.1%, 19.8%, and 17.7%. The method consistently demonstrates strong efficacy across diverse architectures and BERT fine-tuning tasks.

Abstract

Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE), whose forward/backward mismatch produces severe accuracy degradation as networks deepen. We study an orthogonal axis: when and where binarization is enforced during training. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output, using stochastic partial masks with soft refresh. StoMPP delivers two complementary benefits. As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth (ResNet-50 BNN: +18.0/+13.5/+3.8 on CIFAR-10/100/ImageNet), and the pattern holds across ResNet-18/34/50, MobileNetV2, and BERT fine-tuning. Composed with surrogate gradients by applying STE only to frozen entries, it reaches +27.1/+19.8/+17.7 over vanilla STE on the same setting. Underlying both regimes is a single mechanistic finding: progression order is decisive. Forward layerwise progression prevents depth collapse, reverse progression collapses to near-chance, and binary-weight networks (without binary activations) are insensitive to order. We trace this asymmetry to activation-induced gradient blockades: a committed binary activation severs gradient flow upstream, and ordering controls when these blockades form. To isolate the progression's contribution from any benefit conferred by STE, we conduct all ablations in the STE-free regime; the resulting characterization (schedule, refresh, ordering, dynamics) thus reflects the progression itself rather than its interaction with surrogate gradients.

Problem

• Binary Neural Networks
• Straight-Through Estimator
• Depth Scaling
• Training Stability
• Accuracy Degradation

Innovation

• progressive binarization
• binary neural networks
• layerwise training
• gradient blockade
• STE-free training