Understanding Sample Efficiency in Predictive Coding

2026-05-12

AI Summary

This work addresses the lack of theoretical understanding regarding the sample efficiency advantage of predictive coding (PC) over backpropagation (BP). The authors introduce a novel metric termed “target alignment” and establish, for the first time, a theoretical framework for analyzing the learning efficiency of PC. They derive an analytical expression for this metric in deep linear networks and validate their findings through full-trajectory experiments and nonlinear models. Both theoretical and empirical results demonstrate that PC significantly outperforms BP in deep, narrow, and pretrained networks. Moreover, the efficiency of PC remains robust even when certain theoretical assumptions are violated. The study also precisely characterizes the conditions under which optimal target alignment—and thus maximal learning efficiency—is achieved.

Abstract

Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

Problem

• Predictive Coding
• Sample Efficiency
• Target Alignment
• Backpropagation
• Deep Learning

Innovation

• Predictive Coding
• Sample Efficiency
• Target Alignment
• Deep Linear Networks
• Backpropagation