$i$MIND: Insightful Multi-subject Invariant Neural Decoding

2025-09-21

AI Summary

Existing visual neural decoding methods suffer from limited interpretability, hindering insights into perceptual and cognitive mechanisms. To address this, we propose the first multi-subject invariant neural decoding framework that jointly performs biometric identification (e.g., subject identity) and semantic understanding (e.g., stimulus category), thereby constructing a shared neural representational space across subjects. This yields voxel-object activation fingerprints that simultaneously encode object-specific neural responses and subject-specific attentional variability. Our method integrates Vision Transformer-based masked autoencoding, neural feature disentanglement, and multi-subject alignment techniques. Evaluated on multiple fMRI datasets, it achieves state-of-the-art performance. Notably, it enables, for the first time, voxel-level semantic selectivity mapping and cross-subject comparable analysis of neural dynamics—significantly enhancing both decoding interpretability and generalizability.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Decoding visual signals holds the tantalizing potential to unravel the complexities of cognition and perception. While recent studies have focused on reconstructing visual stimuli from neural recordings to bridge brain activity with visual imagery, existing methods offer limited insights into the underlying mechanisms of visual processing in the brain. To mitigate this gap, we present an extit{i}nsightful extbf{M}ulti-subject extbf{I}nvariant extbf{N}eural extbf{D}ecoding ($i$MIND) model, which employs a novel dual-decoding framework--both biometric and semantic decoding--to offer neural interpretability in a data-driven manner and deepen our understanding of brain-based visual functionalities. Our $i$MIND model operates through three core steps: establishing a shared neural representation space across subjects using a ViT-based masked autoencoder, disentangling neural features into complementary subject-specific and object-specific components, and performing dual decoding to support both biometric and semantic classification tasks. Experimental results demonstrate that $i$MIND achieves state-of-the-art decoding performance with minimal scalability limitations. Furthermore, $i$MIND empirically generates voxel-object activation fingerprints that reveal object-specific neural patterns and enable investigation of subject-specific variations in attention to identical stimuli. These findings provide a foundation for more interpretable and generalizable subject-invariant neural decoding, advancing our understanding of the voxel semantic selectivity as well as the neural vision processing dynamics.

Problem

• Decoding visual signals from neural recordings to understand cognition
• Providing neural interpretability through biometric and semantic decoding
• Establishing shared neural representations across multiple human subjects

Innovation

• Dual-decoding framework for biometric and semantic classification
• Shared neural representation space using ViT-based autoencoder
• Disentangling neural features into subject and object components