Towards the Training of Deeper Predictive Coding Neural Networks

2025-06-30

AI Summary

Prediction coding networks (PCNs) suffer significant performance degradation beyond 5–7 layers due to optimization difficulties of latent variables and error accumulation. To address this, we propose a novel deep PCN training framework comprising two core innovations: (1) a precision-weighted latent variable relaxation scheme that dynamically adjusts per-layer optimization step sizes to balance energy distribution across the hierarchy; and (2) an inter-layer guided weight update mechanism that mitigates error propagation from deeper layers. The framework integrates iterative energy-minimization inference with an enhanced balanced propagation algorithm. Evaluated on CIFAR-10/100 and Tiny-ImageNet, our method improves test accuracy by 3.2–5.7 percentage points for PCNs with seven or more layers—marking the first demonstration of performance parity with comparably sized backpropagation-based models. This work establishes a new paradigm for scalable, interpretable, and biologically plausible PCN training.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Predictive coding networks trained with equilibrium propagation are neural models that perform inference through an iterative energy minimization process. Previous studies have demonstrated their effectiveness in shallow architectures, but show significant performance degradation when depth exceeds five to seven layers. In this work, we show that the reason behind this degradation is due to exponentially imbalanced errors between layers during weight updates, and predictions from the previous layer not being effective in guiding updates in deeper layers. We address the first issue by introducing two novel methods to optimize the latent variables that use precision-weighting to re-balance the distribution of energy among layers during the `relaxation phase', and the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers. Empirically, we test our methods on a large number of image classification tasks, resulting in large improvements in test accuracy across networks with more than seven layers, with performances comparable to those of backprop on similar models. These findings suggest that a better understanding of the relaxation phase is important to train models using equilibrium propagation at scale, and open new possibilities for their application in complex tasks.

Problem

• Address performance degradation in deep predictive coding networks
• Balance layer errors during weight updates via precision-weighting
• Reduce error accumulation in deeper layers with new update mechanism

Innovation

• Precision-weighting balances layer energy distribution
• Novel weight update reduces deep layer errors
• Improved training for deeper predictive coding networks