Stochastic Engrams for Efficient Continual Learning with Binarized Neural Networks

2025-03-27

AI Summary

To address catastrophic forgetting-induced performance degradation in Binary Neural Networks (BNNs) under continual learning, this paper proposes a stochastically activated engram-gating mechanism—inspired by neuroscientific memory engram theory—to construct meta-plastic BNNs (mBNNs). Our approach is the first to integrate stochastic engram modeling with meta-plasticity in BNNs, achieving a stable plasticity–stability trade-off while preserving ultra-low computational overhead. Experiments demonstrate: (i) over 20% improvement in average accuracy under class-incremental learning; (ii) domain-incremental performance on par with state-of-the-art full-precision models; and (iii) <5% reduction in GPU peak memory and 20% lower RAM consumption. This work establishes a neurobiologically grounded paradigm for lightweight continual learning, bridging neural mechanisms with efficient binary model adaptation.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The ability to learn continuously in artificial neural networks (ANNs) is often limited by catastrophic forgetting, a phenomenon in which new knowledge becomes dominant. By taking mechanisms of memory encoding in neuroscience (aka. engrams) as inspiration, we propose a novel approach that integrates stochastically-activated engrams as a gating mechanism for metaplastic binarized neural networks (mBNNs). This method leverages the computational efficiency of mBNNs combined with the robustness of probabilistic memory traces to mitigate forgetting and maintain the model's reliability. Previously validated metaplastic optimization techniques have been incorporated to enhance synaptic stability further. Compared to baseline binarized models and benchmark fully connected continual learning approaches, our method is the only strategy capable of reaching average accuracies over 20% in class-incremental scenarios and achieving comparable domain-incremental results to full precision state-of-the-art methods. Furthermore, we achieve a significant reduction in peak GPU and RAM usage, under 5% and 20%, respectively. Our findings demonstrate (A) an improved stability vs. plasticity trade-off, (B) a reduced memory intensiveness, and (C) an enhanced performance in binarized architectures. By uniting principles of neuroscience and efficient computing, we offer new insights into the design of scalable and robust deep learning systems.

Problem

• Mitigate catastrophic forgetting in continual learning
• Enhance synaptic stability in binarized neural networks
• Reduce GPU and RAM usage in deep learning

Innovation

• Stochastic engrams for metaplastic binarized networks
• Combines mBNNs with probabilistic memory traces
• Reduces GPU and RAM usage significantly