Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction

2025-10-25

AI Summary

fMRI-to-image reconstruction suffers from neural information hierarchy collapse and misalignment between neural encoding and generative requirements. Existing diffusion-based approaches rely on a single fixed high-level embedding, leading to semantic distortion and computational inefficiency. This paper introduces MindHier, the first fMRI-driven hierarchical autoregressive image generation framework. It employs a hierarchical fMRI encoder to extract multi-scale neural representations and incorporates a layer-wise alignment mechanism with scale-aware neural guidance, enabling coarse-to-fine, cognitively aligned generation. By integrating CLIP-based feature alignment and hierarchical modeling, MindHier achieves significantly improved semantic fidelity on the Natural Scenes Dataset (NSD), accelerates inference by 4.67× over diffusion baselines, and yields more deterministic outputs—demonstrating comprehensive superiority across all key metrics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reconstructing visual stimuli from fMRI signals is a central challenge bridging machine learning and neuroscience. Recent diffusion-based methods typically map fMRI activity to a single high-level embedding, using it as fixed guidance throughout the entire generation process. However, this fixed guidance collapses hierarchical neural information and is misaligned with the stage-dependent demands of image reconstruction. In response, we propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling. MindHier introduces three components: a Hierarchical fMRI Encoder to extract multi-level neural embeddings, a Hierarchy-to-Hierarchy Alignment scheme to enforce layer-wise correspondence with CLIP features, and a Scale-Aware Coarse-to-Fine Neural Guidance strategy to inject these embeddings into autoregression at matching scales. These designs make MindHier an efficient and cognitively-aligned alternative to diffusion-based methods by enabling a hierarchical reconstruction process that synthesizes global semantics before refining local details, akin to human visual perception. Extensive experiments on the NSD dataset show that MindHier achieves superior semantic fidelity, 4.67x faster inference, and more deterministic results than the diffusion-based baselines.

Problem

• Reconstructing visual stimuli from fMRI signals
• Overcoming limitations of fixed neural guidance in diffusion methods
• Aligning hierarchical neural information with image generation stages

Innovation

• Hierarchical fMRI Encoder extracts multi-level neural embeddings
• Hierarchy-to-Hierarchy Alignment enforces layer-wise CLIP correspondence
• Scale-Aware Coarse-to-Fine Guidance injects embeddings at matching scales