BD-Net: Has Depth-Wise Convolution Ever Been Applied in Binary Neural Networks?

2025-11-19

AI Summary

To address the weak representational capacity and training instability of depthwise separable convolutions in Binary Neural Networks (BNNs) caused by aggressive binarization, this paper introduces two key innovations: (1) 1.58-bit convolution, which combines partial weight ternarization (±1, 0) with binary activations to significantly enhance the expressive power of depthwise convolutions under ultra-low-bit constraints; and (2) pre-BatchNorm residual connections, which place BatchNorm before the residual branch and optimize gradient flow to improve the Hessian condition number and training stability. Integrated into MobileNetV1, the proposed method achieves only 33M OPs on ImageNet while attaining a Top-1 accuracy up to 9.3 percentage points higher than prior BNNs—setting a new state-of-the-art. Moreover, it consistently outperforms existing methods across multiple benchmarks, including CIFAR-10/100, STL-10, and Tiny ImageNet.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advances in model compression have highlighted the potential of low-bit precision techniques, with Binary Neural Networks (BNNs) attracting attention for their extreme efficiency. However, extreme quantization in BNNs limits representational capacity and destabilizes training, posing significant challenges for lightweight architectures with depth-wise convolutions. To address this, we propose a 1.58-bit convolution to enhance expressiveness and a pre-BN residual connection to stabilize optimization by improving the Hessian condition number. These innovations enable, to the best of our knowledge, the first successful binarization of depth-wise convolutions in BNNs. Our method achieves 33M OPs on ImageNet with MobileNet V1, establishing a new state-of-the-art in BNNs by outperforming prior methods with comparable OPs. Moreover, it consistently outperforms existing methods across various datasets, including CIFAR-10, CIFAR-100, STL-10, Tiny ImageNet, and Oxford Flowers 102, with accuracy improvements of up to 9.3 percentage points.

Problem

• Binarizing depth-wise convolutions in Binary Neural Networks
• Overcoming representational limitations from extreme quantization
• Stabilizing training dynamics in lightweight binary architectures

Innovation

• 1.58-bit convolution enhances BNN expressiveness
• Pre-BN residual connection stabilizes optimization process
• First successful binarization of depth-wise convolutions