NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery

2025-06-07

AI Summary

Current fMRI-based visual decoding models exhibit poor generalization to mental imagery reconstruction due to low signal-to-noise ratio and coarse spatial resolution of imagery-related neural activity, compounded by their exclusive optimization for perceived (“seen”) stimuli. Method: We introduce NSD-Imagery—the first large-scale, paired fMRI–mental imagery benchmark dataset—designed to fill the critical evaluation gap in mind-reading image reconstruction. We conduct systematic transfer evaluations and architectural comparisons across state-of-the-art models (MindEye1/2, Brain Diffuser, iCNN). Contribution/Results: Our analysis reveals a novel performance decoupling phenomenon: linear decoders with multimodal feature fusion consistently outperform complex end-to-end architectures, challenging the prevailing complexity-driven paradigm. We further demonstrate that NSD-pretrained models suffer consistent degradation in imagery reconstruction, establishing a new lightweight, interpretable, and cross-modal generalizable decoding framework. NSD-Imagery provides a reproducible benchmark and methodological foundation for brain–computer interfaces and clinical neural decoding.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We release NSD-Imagery, a benchmark dataset of human fMRI activity paired with mental images, to complement the existing Natural Scenes Dataset (NSD), a large-scale dataset of fMRI activity paired with seen images that enabled unprecedented improvements in fMRI-to-image reconstruction efforts. Recent models trained on NSD have been evaluated only on seen image reconstruction. Using NSD-Imagery, it is possible to assess how well these models perform on mental image reconstruction. This is a challenging generalization requirement because mental images are encoded in human brain activity with relatively lower signal-to-noise and spatial resolution; however, generalization from seen to mental imagery is critical for real-world applications in medical domains and brain-computer interfaces, where the desired information is always internally generated. We provide benchmarks for a suite of recent NSD-trained open-source visual decoding models (MindEye1, MindEye2, Brain Diffuser, iCNN, Takagi et al.) on NSD-Imagery, and show that the performance of decoding methods on mental images is largely decoupled from performance on vision reconstruction. We further demonstrate that architectural choices significantly impact cross-decoding performance: models employing simple linear decoding architectures and multimodal feature decoding generalize better to mental imagery, while complex architectures tend to overfit visual training data. Our findings indicate that mental imagery datasets are critical for the development of practical applications, and establish NSD-Imagery as a useful resource for better aligning visual decoding methods with this goal.

Problem

• Extends fMRI vision decoding to mental imagery tasks
• Evaluates model performance on mental image reconstruction
• Assesses architectural impact on cross-decoding generalization

Innovation

• Benchmark dataset for mental imagery fMRI decoding
• Linear and multimodal models generalize better
• Performance decoupled from vision reconstruction