Towards Interpretable Visual Decoding with Attention to Brain Representations
Existing fMRI-to-image decoding methods typically rely on intermediate feature spaces (e.g., image or text embeddings), obscuring the dynamic, region-specific contributions of cortical areas to the generative process. To address this, we propose NeuroAdapter—a novel framework that enables end-to-end, direct conditioning of latent diffusion models (LDMs) on fMRI signals, bypassing intermediate representations to preserve neural information fidelity. Furthermore, we introduce IBBI (Interpretable Bidirectional Brain–Image), a bidirectional interpretability framework that quantitatively characterizes the spatiotemporal modulation of generation stages by distinct brain regions through analysis of cross-attention weight distributions across diffusion timesteps. Evaluated on public fMRI datasets, our method achieves reconstruction quality competitive with state-of-the-art approaches while substantially enhancing the interpretability of neural–image correspondences. This work establishes a new paradigm for brain–computer interfaces and computational neuroscience by unifying high-fidelity neural decoding with mechanistic, process-level interpretability.