Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs

2025-08-21

AI Summary

Balancing propagation (EP) in deep convolutional recurrent neural networks (CRNNs) suffers from vanishing gradients, hindering convergence. To address this, we propose a novel EP training framework integrating hierarchical local error signals, biologically plausible local learning rules, and knowledge distillation. Specifically, intermediate error signals are injected into the two-phase energy minimization process to enhance information flow and dynamical stability of deep neurons. Moreover, knowledge distillation is introduced—*for the first time*—into the EP paradigm to improve gradient estimation fidelity. This design overcomes EP’s traditional limitation to shallow architectures, substantially improving its scalability to deep networks. Experiments demonstrate state-of-the-art performance on CIFAR-10 and CIFAR-100, and successful training of deep VGG models, validating both effectiveness and practical applicability.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Equilibrium Propagation (EP) is a biologically inspired local learning rule first proposed for convergent recurrent neural networks (CRNNs), in which synaptic updates depend only on neuron states from two distinct phases. EP estimates gradients that closely align with those computed by Backpropagation Through Time (BPTT) while significantly reducing computational demands, positioning it as a potential candidate for on-chip training in neuromorphic architectures. However, prior studies on EP have been constrained to shallow architectures, as deeper networks suffer from the vanishing gradient problem, leading to convergence difficulties in both energy minimization and gradient computation. To address the vanishing gradient problem in deep EP networks, we propose a novel EP framework that incorporates intermediate error signals to enhance information flow and convergence of neuron dynamics. This is the first work to integrate knowledge distillation and local error signals into EP, enabling the training of significantly deeper architectures. Our proposed approach achieves state-of-the-art performance on the CIFAR-10 and CIFAR-100 datasets, showcasing its scalability on deep VGG architectures. These results represent a significant advancement in the scalability of EP, paving the way for its application in real-world systems.

Problem

• Addresses vanishing gradients in deep Equilibrium Propagation networks
• Enables scalable on-chip training for neuromorphic architectures
• Improves convergence in deep convolutional recurrent neural networks

Innovation

• Intermediate error signals enhance deep EP training
• Knowledge distillation integrated into Equilibrium Propagation
• Scalable EP framework for deep convolutional CRNNs