Neuro-3D: Towards 3D Visual Decoding from EEG Signals

2024-11-19

arXiv.org

AI Summary

This study addresses the challenge of decoding human neural representations of 3D visual stimuli from electroencephalography (EEG) signals. We propose the first EEG-to-3D decoding framework supporting both image and video inputs, covering 72 categories of 3D objects, and enabling end-to-end reconstruction of colored point clouds directly from raw EEG. Methodologically: (1) an adaptive fusion mechanism jointly models static topological and dynamic time-frequency EEG features; (2) a cross-modal representation alignment module bridges neural activity with 3D geometric–chromatic space; and (3) a diffusion model–driven generative decoder enables high-fidelity synthesis. Contributions include: releasing the first open-source, multi-subject EEG-3D benchmark dataset; achieving high-fidelity reconstruction of both 3D shape and color; and uncovering distinct neural mechanisms—occipital dominance in depth encoding and parietal specialization in spatial structural integration—underlying 3D perception.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: 3D Computer Vision
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Human's perception of the visual world is shaped by the stereo processing of 3D information. Understanding how the brain perceives and processes 3D visual stimuli in the real world has been a longstanding endeavor in neuroscience. Towards this goal, we introduce a new neuroscience task: decoding 3D visual perception from EEG signals, a neuroimaging technique that enables real-time monitoring of neural dynamics enriched with complex visual cues. To provide the essential benchmark, we first present EEG-3D, a pioneering dataset featuring multimodal analysis data and extensive EEG recordings from 12 subjects viewing 72 categories of 3D objects rendered in both videos and images. Furthermore, we propose Neuro-3D, a 3D visual decoding framework based on EEG signals. This framework adaptively integrates EEG features derived from static and dynamic stimuli to learn complementary and robust neural representations, which are subsequently utilized to recover both the shape and color of 3D objects through the proposed diffusion-based colored point cloud decoder. To the best of our knowledge, we are the first to explore EEG-based 3D visual decoding. Experiments indicate that Neuro-3D not only reconstructs colored 3D objects with high fidelity, but also learns effective neural representations that enable insightful brain region analysis. The dataset and associated code will be made publicly available.

Problem

• Decoding 3D visual perception from EEG signals
• Reconstructing 3D objects from neural representations
• Analyzing brain regions via EEG-based visual decoding

Innovation

• Decoding 3D visual perception from EEG signals
• Multimodal EEG dataset with 3D object stimuli
• Diffusion-based colored point cloud decoder