FreqSelect: Frequency-Aware fMRI-to-Image Reconstruction

2025-05-18

AI Summary

Natural image reconstruction from fMRI is challenging due to high noise levels and low spatial resolution, while existing models struggle to capture full-spectrum neural representations. To address this, we propose a frequency-aware end-to-end reconstruction framework. Our key contributions are: (1) FreqSelect—a novel lightweight, unsupervised dynamic frequency selection module that enables content-adaptive band gating and interpretable modeling of neural frequency preferences; and (2) a hybrid architecture integrating VAE and diffusion models with a frequency-domain adaptive filtering mechanism, enabling fully end-to-end training without auxiliary annotations. Evaluated on the Natural Scenes dataset, our method achieves significant improvements in reconstruction fidelity—simultaneously enhancing both low-level structural accuracy (e.g., PSNR, SSIM) and high-level semantic consistency (e.g., CLIP score). It further demonstrates strong cross-subject and cross-scene generalization and is readily extensible to other neuroimaging modalities.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reconstructing natural images from functional magnetic resonance imaging (fMRI) data remains a core challenge in natural decoding due to the mismatch between the richness of visual stimuli and the noisy, low resolution nature of fMRI signals. While recent two-stage models, combining deep variational autoencoders (VAEs) with diffusion models, have advanced this task, they treat all spatial-frequency components of the input equally. This uniform treatment forces the model to extract meaning features and suppress irrelevant noise simultaneously, limiting its effectiveness. We introduce FreqSelect, a lightweight, adaptive module that selectively filters spatial-frequency bands before encoding. By dynamically emphasizing frequencies that are most predictive of brain activity and suppressing those that are uninformative, FreqSelect acts as a content-aware gate between image features and natural data. It integrates seamlessly into standard very deep VAE-diffusion pipelines and requires no additional supervision. Evaluated on the Natural Scenes dataset, FreqSelect consistently improves reconstruction quality across both low- and high-level metrics. Beyond performance gains, the learned frequency-selection patterns offer interpretable insights into how different visual frequencies are represented in the brain. Our method generalizes across subjects and scenes, and holds promise for extension to other neuroimaging modalities, offering a principled approach to enhancing both decoding accuracy and neuroscientific interpretability.

Problem

• Reconstructing images from noisy fMRI data effectively
• Addressing uniform treatment of spatial-frequency components in models
• Enhancing decoding accuracy and neuroscientific interpretability

Innovation

• Frequency-aware adaptive module for fMRI filtering
• Seamless integration into VAE-diffusion pipelines
• Dynamic emphasis on predictive brain activity frequencies