🤖 AI Summary
Existing neural decoding methods rely on pre-trained image or text feature vectors, yet their semantic structures fundamentally mismatch the brain’s intrinsic neural representations, limiting decoding accuracy. To address this, we propose Brain Alignment—a novel framework that explicitly aligns semantic vector spaces with cortical functional organization via fMRI-supervised fine-tuning. This approach bridges the modality gap by learning brain-grounded semantic embeddings. Crucially, Brain Alignment enables zero-shot transfer to MEG and ECoG modalities, significantly improving stimulus reconstruction correlation (p < 0.001) and fine-grained semantic category classification accuracy across modalities. The gains are robust, generalizable, and reproducible. Furthermore, our analysis reveals that the choice of source semantic space critically determines alignment efficacy—highlighting the importance of architectural compatibility between pretrained features and neural dynamics. This work establishes a principled, brain-informed paradigm for cross-modal neural decoding.
📝 Abstract
The development of algorithms to accurately decode neural information has long been a research focus in the field of neuroscience. Brain decoding typically involves training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might fundamentally differ from those that are encoded by the brain, limiting the ability of decoders to accurately learn this mapping. To address this issue, we propose a framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the brain. We trained this model with functional magnetic resonance imaging (fMRI) and then performed zero-shot brain decoding on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data. fMRI-based brain-aligned vectors improved decoding performance across all three neuroimaging datasets when accuracy was determined by calculating the correlation coefficients between true and predicted vectors. Additionally, when decoding accuracy was determined via stimulus identification, this accuracy increased in specific category types; improvements varied depending on the original vector space that was used for brain-alignment, and consistent improvements were observed across all neuroimaging modalities.