Cross-Modal Alignment between Visual Stimuli and Neural Responses in the Visual Cortex

2025-11-06

AI Summary

Stimulus–neural response mappings in visual cortex are highly susceptible to neural variability and recording noise, leading to overfitting and poor generalization in conventional generative encoding–decoding frameworks. Method: This paper proposes Visual–Neural Alignment (VNA), a discriminative cross-modal alignment framework that replaces traditional generative paradigms by jointly optimizing encoder and decoder objectives to explicitly align visual representations with neural response spaces. Results: Evaluated on multiple invasive cortical datasets from mice and macaques, VNA consistently outperforms state-of-the-art direct mapping methods in both neural encoding (stimulus-to-response prediction) and stimulus decoding (response-to-stimulus reconstruction), achieving average improvements of 12.7%–23.4%. Moreover, VNA demonstrates superior generalization across subjects and experimental conditions, effectively mitigating overfitting induced by neural noise and inter-trial variability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Investigating the mapping between visual stimuli and neural responses in the visual cortex contributes to a deeper understanding of biological visual processing mechanisms. Most existing studies characterize this mapping by training models to directly encode visual stimuli into neural responses or decode neural responses into visual stimuli. However, due to neural response variability and limited neural recording techniques, these studies suffer from overfitting and lack generalizability. Motivated by this challenge, in this paper we shift the tasks from conventional direct encoding and decoding to discriminative encoding and decoding, which are more reasonable. And on top of this we propose a cross-modal alignment approach, named Visual-Neural Alignment (VNA). To thoroughly test the performance of the three methods (direct encoding, direct decoding, and our proposed VNA) on discriminative encoding and decoding tasks, we conduct extensive experiments on three invasive visual cortex datasets, involving two types of subject mammals (mice and macaques). The results demonstrate that our VNA generally outperforms direct encoding and direct decoding, indicating our VNA can most precisely characterize the above visual-neural mapping among the three methods.

Problem

• Mapping visual stimuli to neural responses in visual cortex
• Overcoming neural variability and recording limitations
• Developing cross-modal alignment for visual-neural mapping

Innovation

• Shifted tasks to discriminative encoding and decoding
• Proposed cross-modal alignment approach named VNA
• Outperformed direct encoding and decoding methods