BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Training

2025-12-03

AI Summary

Binary neural networks (BNNs) suffer from reliance on floating-point arithmetic during backpropagation, hindering fully binary training. To address this, we propose Binary Error Propagation (BEP), the first method to formulate a discrete-chain rule for error propagation using purely binary vectors to represent gradient signals. BEP performs both forward and backward passes exclusively via bitwise operations—eliminating all floating-point computations and full-precision parameters. This enables end-to-end fully binary training, with particular efficacy for recurrent neural network (RNN) architectures. Experiments demonstrate significant accuracy improvements: +6.89% on multilayer perceptrons and +10.57% on RNNs, respectively, over baseline BNNs. The implementation is publicly available.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Binary Neural Networks (BNNs), which constrain both weights and activations to binary values, offer substantial reductions in computational complexity, memory footprint, and energy consumption. These advantages make them particularly well suited for deployment on resource-constrained devices. However, training BNNs via gradient-based optimization remains challenging due to the discrete nature of their variables. The dominant approach, quantization-aware training, circumvents this issue by employing surrogate gradients. Yet, this method requires maintaining latent full-precision parameters and performing the backward pass with floating-point arithmetic, thereby forfeiting the efficiency of binary operations during training. While alternative approaches based on local learning rules exist, they are unsuitable for global credit assignment and for back-propagating errors in multi-layer architectures. This paper introduces Binary Error Propagation (BEP), the first learning algorithm to establish a principled, discrete analog of the backpropagation chain rule. This mechanism enables error signals, represented as binary vectors, to be propagated backward through multiple layers of a neural network. BEP operates entirely on binary variables, with all forward and backward computations performed using only bitwise operations. Crucially, this makes BEP the first solution to enable end-to-end binary training for recurrent neural network architectures. We validate the effectiveness of BEP on both multi-layer perceptrons and recurrent neural networks, demonstrating gains of up to +6.89% and +10.57% in test accuracy, respectively. The proposed algorithm is released as an open-source repository.

Problem

• Training Binary Neural Networks is challenging due to discrete variables
• Existing methods lose efficiency by using floating-point arithmetic in training
• Current approaches fail to enable end-to-end binary training for recurrent networks

Innovation

• Binary Error Propagation enables backpropagation with binary signals
• Uses only bitwise operations for all forward and backward computations
• Enables end-to-end binary training for recurrent neural networks