Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI

2025-05-20

AI Summary

Existing brain-to-image decoding methods rely on multi-stage pipelines and preprocessing—particularly temporal compression of fMRI signals—hindering high-temporal-resolution image reconstruction. This work introduces the first single-stage, end-to-end diffusion model that directly synthesizes images from raw time-series fMRI data, eliminating dimensionality reduction and sequential processing. Key contributions include: (1) a temporally aware fMRI feature encoder that explicitly models the dynamic evolution of neural activity; and (2) a cross-modal latent-space alignment mechanism coupled with dynamic conditional generation, enabling both semantic-level image reconstruction and fine-grained temporal representation disentanglement. Evaluated on dynamic fMRI datasets, our method significantly outperforms state-of-the-art approaches—especially on semantic similarity metrics—while maintaining competitive performance on static fMRI benchmarks. Notably, it achieves the first millisecond-scale characterization of evolving image representations directly decoded from neural dynamics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Brain-to-image decoding has been recently propelled by the progress in generative AI models and the availability of large ultra-high field functional Magnetic Resonance Imaging (fMRI). However, current approaches depend on complicated multi-stage pipelines and preprocessing steps that typically collapse the temporal dimension of brain recordings, thereby limiting time-resolved brain decoders. Here, we introduce Dynadiff (Dynamic Neural Activity Diffusion for Image Reconstruction), a new single-stage diffusion model designed for reconstructing images from dynamically evolving fMRI recordings. Our approach offers three main contributions. First, Dynadiff simplifies training as compared to existing approaches. Second, our model outperforms state-of-the-art models on time-resolved fMRI signals, especially on high-level semantic image reconstruction metrics, while remaining competitive on preprocessed fMRI data that collapse time. Third, this approach allows a precise characterization of the evolution of image representations in brain activity. Overall, this work lays the foundation for time-resolved brain-to-image decoding.

Problem

• Simplifies training for fMRI-based image reconstruction
• Improves time-resolved fMRI decoding accuracy
• Enables tracking image representation evolution in brain activity

Innovation

• Single-stage diffusion model for fMRI decoding
• Simplifies training compared to multi-stage pipelines
• Enhances time-resolved semantic image reconstruction