Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment

2026-02-17

AI Summary

This work addresses the limitations of predictive coding (PC) in deep networks, where layer-wise backpropagation of error signals incurs substantial latency and gradient attenuation. To overcome these issues, the authors propose Direct Kolen-Pollack Predictive Coding (DKP-PC), which integrates direct feedback alignment with the Kolen-Pollack learning rule into the PC framework for the first time. DKP-PC introduces learnable direct feedback pathways from the output layer to each hidden layer, enabling local weight updates and reducing the time complexity of error propagation from O(L) to O(1). This eliminates depth-dependent delays and mitigates gradient decay. Experimental results demonstrate that DKP-PC matches or exceeds the performance of standard PC while significantly reducing training latency and computational overhead, highlighting its potential for deployment on specialized hardware.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates, allowing parallel learning across layers. However, practical implementations face two key limitations: error signals must still propagate from the output to early layers through multiple inference-phase steps, and feedback decays exponentially during this process, leading to vanishing updates in early layers. We propose direct Kolen-Pollack predictive coding (DKP-PC), which simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC while preserving update locality. Leveraging direct feedback alignment and direct Kolen-Pollack algorithms, DKP-PC introduces learnable feedback connections from the output layer to all hidden layers, establishing a direct pathway for error transmission. This yields an algorithm that reduces the theoretical error propagation time complexity from O(L), with L being the network depth, to O(1), removing depth-dependent delay in error signals. Moreover, empirical results demonstrate that DKP-PC achieves performance at least comparable to, and often exceeding, that of standard PC, while offering improved latency and computational performance, supporting its potential for custom hardware-efficient implementations.

Problem

• Predictive Coding
• Feedback Delay
• Exponential Decay
• Vanishing Updates
• Error Propagation

Innovation

• Predictive Coding
• Direct Feedback Alignment
• Kolen-Pollack Algorithm
• Local Learning
• Error Propagation