Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks

2026-07-14

AI Summary

Predictive coding (PC) has faced challenges in biological plausibility and strict locality due to its reliance on non-local automatic differentiation involving Jacobian transposes. This work proposes WF-Act-PC, the first method to exactly implement local Jacobian transposes in modern networks incorporating normalization and activation functions. By decomposing the transpose into three locally computable products, and under conditions such as weight symmetry, the approach enables fully local error signal propagation without dependence on backpropagation. Integrating weight feedback, gain modeling of normalization layers, local estimation of activation derivatives, and an approximation for MaxPool, WF-Act-PC significantly outperforms iPC on CIFAR-10/100 and Tiny-ImageNet, achieving accuracy on par with or exceeding tuned backpropagation in deep architectures such as VGG-9 and ResNet-18.

Abstract

Predictive Coding (PC) offers a biologically motivated alternative to backpropagation via local weight updates, yet routing error between layers still relies on an autograd Jacobian-transpose ($J^\top$) product - the last non-local operation in PC. We show that this dependency is largely avoidable. For any layer $f(x)=\mathrm{Act}(\mathrm{Norm}(L(x)))$ with frozen normalization statistics, the exact $J^\top$ factors into three locally available terms, $J^\top v = L^\top(s \odot σ'(z) \odot v)$, where $σ'$ is the activation derivative, $z$ is the pre-activation, and $s=γ/σ_{\mathrm{run}}$ is the normalization gain. Prior weight-feedback methods omitted both corrections; restoring them closes the transport gap for this layer class. Locality here holds up to three assumptions, which we state upfront: weight symmetry ($L^\top$ mirrors the forward operator, as assumed by all PC), a soft spectral-norm control that is not synapse-local, and a nearest-neighbour approximation for MaxPool. Substituting the identity into PC yields WF-Act-PC, which removes the autograd backward pass from error transport. On CIFAR-10/100 (50 epochs, 5 seeds), WF-Act-PC is the only PC method whose accuracy improves with depth, surpassing iPC - the strongest classical PC baseline - by 2.7-22.3 pp on CIFAR-10. With both methods tuned per architecture, it matches or exceeds a comparably-tuned backpropagation baseline on the deeper CIFAR-10 architectures (VGG-9: 93.57% vs. 92.43%; ResNet-18: 92.76% vs. 91.54%) and on the harder Tiny-ImageNet benchmark, while trailing tuned BP on the deeper CIFAR-100 VGG cells. Our WF-Act-PC implementation is publicly available at https://github.com/jlshen025/pcax

Problem

• Predictive Coding
• Jacobian Transpose
• Local Learning
• Backpropagation Alternative
• Weight Feedback

Innovation

• Predictive Coding
• Jacobian Transpose
• Local Learning
• Weight Feedback
• Biologically Plausible Backpropagation