Faster Predictive Coding Networks via Better Initialization

2026-01-28

AI Summary

This work addresses the high computational cost and slow convergence of predictive coding networks, which stem from their iterative inference mechanism and hinder practical deployment. To mitigate this limitation, the authors propose an intelligent initialization strategy that leverages the iterative state from the previous training sample to initialize the current neuronal activities. This approach substantially reduces the number of iterations required per forward–backward inference loop while preserving the biological plausibility inherent to predictive coding. By doing so, it effectively narrows the gap between predictive coding and backpropagation in terms of both training efficiency and performance. Experimental results demonstrate that the proposed method achieves faster convergence and lower test loss across both supervised and unsupervised learning tasks, thereby enhancing the practical utility of predictive coding networks.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Probabilistic Circuits and Graphical Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Research aimed at scaling up neuroscience inspired learning algorithms for neural networks is accelerating. Recently, a key research area has been the study of energy-based learning algorithms such as predictive coding, due to their versatility and mathematical grounding. However, the applicability of such methods is held back by the large computational requirements caused by their iterative nature. In this work, we address this problem by showing that the choice of initialization of the neurons in a predictive coding network matters significantly and can notably reduce the required training times. Consequently, we propose a new initialization technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples. Our approach suggests a promising path toward reconciling the disparities between predictive coding and backpropagation in terms of computational efficiency and final performance. In fact, our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.

Problem

• predictive coding
• computational efficiency
• training time
• initialization
• energy-based learning

Innovation

• predictive coding
• initialization
• energy-based learning
• convergence speed
• neural networks