Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images

2026-05-27

AI Summary

This study investigates whether the backpropagation algorithm has a biologically plausible counterpart in the human visual system, with particular focus on whether its gradient signals align with the brain’s hierarchical spatiotemporal responses to natural images. For the first time, we systematically compare the backpropagation-derived gradients from eight self-supervised vision models—including DINOv3—with human neural activity recorded via fMRI and MEG, using spatiotemporal alignment analyses. Our results demonstrate that although these gradient signals can predict neural responses in higher visual areas and at later processing stages, their computational sequence and spatial distribution substantially diverge from the brain’s actual hierarchical processing architecture. This discrepancy reveals a fundamental difference in learning mechanisms between artificial neural networks and biological vision systems.

Abstract

Backpropagation is the core learning mechanism underlying deep learning. However, whether and how this algorithm is implemented in the brain remains highly debated. In particular, while forward activations of pretrained models reliably map onto the cortical hierarchy of visual processing, it is unknown whether backpropagated gradients exhibit a similar correspondence. Here, we address this question using functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) recordings of human brain responses to natural images. For this, we extend standard encoding analyses of forward activations to map backpropagated gradients onto neural data. Focusing on a recent self-supervised vision model (DINOv3) and reproducing results on eight vision models, we find that backpropagated gradients can reliably predict both fMRI and MEG signals, specifically in higher-level visual cortex and for later latencies. However, the spatial and temporal organization of these backpropagated gradients in the brain diverges from the patterns expected under a biologically plausible backpropagation mechanism: specifically, both the order in which gradients are computed and their spatial organization diverge from the temporal and spatial hierarchies of the human brain. Together, these results suggest that, although deep networks and the brain may share similar representational content, they likely rely on fundamentally different mechanisms to learn those representations.

Problem

• backpropagation
• brain hierarchy
• visual cortex
• neural gradients
• representational alignment

Innovation

• backpropagation
• neural encoding
• fMRI
• MEG
• visual hierarchy