Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

2024-11-28

arXiv.org

AI Summary

Existing binary neural networks (BNNs) rely on floating-point stochastic gradient descent (SGD) for training, hindering full exploitation of binary computation benefits. This paper proposes the first fully binary, gradient-free, end-to-end training algorithm—operating exclusively with XNOR, population-count (Popcount), and integer increment/decrement operations—supporting binary inputs, weights, and activations. We introduce a novel update mechanism driven by locally computed binary error signals and incorporate integer-valued hidden weights to model synaptic plasticity, enhancing neurobiological plausibility. On multi-class benchmarks, our method achieves up to 35.47% higher accuracy than prior single-layer fully binary state-of-the-art methods. Compared to full-precision SGD under identical memory constraints, it improves accuracy by 41.31% while reducing computational cost by two orders of magnitude.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Hardware-aware ML
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Binary Neural Networks (BNNs) significantly reduce computational complexity and memory usage in machine and deep learning by representing weights and activations with just one bit. However, most existing training algorithms for BNNs rely on quantization-aware floating-point Stochastic Gradient Descent (SGD), limiting the full exploitation of binary operations to the inference phase only. In this work, we propose, for the first time, a fully binary and gradient-free training algorithm for multi-layer BNNs, eliminating the need for back-propagated floating-point gradients. Specifically, the proposed algorithm relies on local binary error signals and binary weight updates, employing integer-valued hidden weights that serve as a synaptic metaplasticity mechanism, thereby enhancing its neurobiological plausibility. The fully binary and gradient-free algorithm introduced in this paper enables the training of binary multi-layer perceptrons with binary inputs, weights, and activations, by using exclusively XNOR, Popcount, and increment/decrement operations. Experimental results on multi-class classification benchmarks show test accuracy improvements of up to +35.47% over the only existing fully binary single-layer state-of-the-art solution. Compared to full-precision SGD, our solution improves test accuracy by up to +41.31% under the same total memory demand$unicode{x2013}$including the model, activations, and input dataset$unicode{x2013}$while also reducing computational cost by two orders of magnitude in terms of the total number of equivalent Boolean gates. The proposed algorithm is made available to the scientific community as a public repository.

Problem

• Develop fully binary gradient-free training for multi-layer BNNs
• Replace floating-point SGD with local binary error signals
• Improve accuracy and reduce computational cost in BNNs

Innovation

• Fully binary gradient-free training algorithm
• Local binary error signals updates
• XNOR Popcount increment decrement operations