Error Highways: Scaling Predictive Coding to Very Deep Networks

2026-06-21

AI Summary

This work addresses the challenge of training deep predictive coding networks, which suffer from exponentially decaying error signals with increasing depth. To overcome this limitation, the authors propose an error highway mechanism that introduces feedback matrices directly connecting the output layer to each hidden layer within the energy function, thereby establishing depth-independent pathways for error propagation. This approach preserves local synaptic update rules while effectively circumventing the Jacobian chain decay problem inherent in conventional backpropagation through depth. Notably, it enables, for the first time, end-to-end training of predictive coding networks with over one hundred layers. Empirical results demonstrate successful training of 128-layer multilayer perceptrons on MNIST and Fashion-MNIST benchmarks, with performance exhibiting strong robustness to variations in network depth.

Abstract

Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices $V_{L\to i}$ that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the $i$-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Problem

• predictive coding
• error propagation
• deep networks
• learning signal decay
• scaling

Innovation

• predictive coding
• highway error propagation
• very deep networks
• local learning
• error signal attenuation