MinD-3D++: Advancing fMRI-Based 3D Reconstruction with High-Quality Textured Mesh Generation and a Comprehensive Dataset

2024-09-17

AI Summary

This study addresses the challenge of high-fidelity reconstruction of 3D object geometry and surface texture directly from fMRI signals. We propose a multimodal encoder-decoder framework integrating an fMRI feature alignment module, Neural Radiance Fields (NeRF), differentiable rendering, and CLIP-driven cross-modal supervision. Our approach achieves, for the first time, end-to-end generation of textured 3D meshes from fMRI. We introduce the first large-scale fMRI-3D dataset—comprising 117 object categories, 4,768 objects, and an fMRI-Objaverse subset—enabling cross-subject and out-of-distribution 3D mental reconstruction. We further establish a three-tier evaluation benchmark covering semantic correctness, structural fidelity, and textural realism. On fMRI-3D, our method improves semantic accuracy by 32.7% and reduces texture FID by 41.5%. Neuroscientific analysis reveals a functional dissociation: occipital cortex predominantly encodes 3D shape, while parietal cortex exhibits specificity for surface texture representation.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI) data, introduced as Recon3DMind, is of significant interest to both cognitive neuroscience and computer vision. To advance this task, we present the fMRI-3D dataset, which includes data from 15 participants and showcases a total of 4,768 3D objects. The dataset consists of two components: fMRI-Shape, previously introduced and available at https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse, proposed in this paper and available at https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse includes data from 5 subjects, 4 of whom are also part of the core set in fMRI-Shape. Each subject views 3,142 3D objects across 117 categories, all accompanied by text captions. This significantly enhances the diversity and potential applications of the dataset. Moreover, we propose MinD-3D++, a novel framework for decoding textured 3D visual information from fMRI signals. The framework evaluates the feasibility of not only reconstructing 3D objects from the human mind but also generating, for the first time, 3D textured meshes with detailed textures from fMRI data. We establish new benchmarks by designing metrics at the semantic, structural, and textured levels to evaluate model performance. Furthermore, we assess the model's effectiveness in out-of-distribution settings and analyze the attribution of the proposed 3D pari fMRI dataset in visual regions of interest (ROIs) in fMRI signals. Our experiments demonstrate that MinD-3D++ not only reconstructs 3D objects with high semantic and spatial accuracy but also provides deeper insights into how the human brain processes 3D visual information. Project page: https://jianxgao.github.io/MinD-3D.

Problem

• fMRI technology
• 3D image reconstruction
• color and texture information

Innovation

• MinD-3D++
• 3D reconstruction
• fMRI decoding