Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

2025-03-14

AI Summary

This study addresses the challenge of high-fidelity dynamic video reconstruction from fMRI signals, specifically tackling insufficient spatiotemporal coherence and semantic accuracy. We propose the first multi-task collaborative framework grounded in functional specialization of the visual cortex, decomposing reconstruction into four sequential subtasks: key-object segmentation, concept identification, scene description, and blurry video generation—whose outputs jointly serve as conditional signals to guide a pre-trained text-to-video diffusion model (e.g., Stable Video Diffusion). Our method integrates fMRI encoding modeling, multi-granularity visual understanding, and neuro-prior-guided generation to achieve dual alignment between neural representations and both video semantics and temporal structure. Experiments demonstrate a 26.6% improvement in video temporal consistency and a 19.1% gain in semantic accuracy over state-of-the-art methods. The framework further offers intrinsic interpretability and promising potential for brain–computer interface applications.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Decoding visual stimuli from neural activity is essential for understanding the human brain. While fMRI methods have successfully reconstructed static images, fMRI-to-video reconstruction faces challenges due to the need for capturing spatiotemporal dynamics like motion and scene transitions. Recent approaches have improved semantic and perceptual alignment but struggle to integrate coarse fMRI data with detailed visual features. Inspired by the hierarchical organization of the visual system, we propose NEURONS, a novel framework that decouples learning into four correlated sub-tasks: key object segmentation, concept recognition, scene description, and blurry video reconstruction. This approach simulates the visual cortex's functional specialization, allowing the model to capture diverse video content. In the inference stage, NEURONS generates robust conditioning signals for a pre-trained text-to-video diffusion model to reconstruct the videos. Extensive experiments demonstrate that NEURONS outperforms state-of-the-art baselines, achieving solid improvements in video consistency (26.6%) and semantic-level accuracy (19.1%). Notably, NEURONS shows a strong functional correlation with the visual cortex, highlighting its potential for brain-computer interfaces and clinical applications. Code and model weights will be available at: https://github.com/xmed-lab/NEURONS.

Problem

• Decoding visual stimuli from fMRI data for video reconstruction.
• Integrating coarse fMRI data with detailed visual features.
• Improving video consistency and semantic accuracy in fMRI-to-video tasks.

Innovation

• Hierarchical framework simulating visual cortex functions
• Decouples learning into four specialized sub-tasks
• Uses pre-trained diffusion model for video reconstruction