Achieving more human brain-like vision via human EEG representational alignment

2024-01-30

arXiv.org

AI Summary

Current vision models exhibit substantial discrepancies from human visual cortex information processing, primarily due to the absence of high-quality, non-invasive human brain response data for representation alignment. To address this, we propose ReAlnet—the first vision model grounded in multi-layer EEG–image cross-modal alignment. Our method introduces: (1) the first end-to-end, layer-wise alignment from image features to non-invasive EEG responses; (2) an optimization-enabled encoding framework supporting generalization across network layers, object categories, and modalities; and (3) a data-driven training paradigm integrating neural encoding modeling, joint representation learning, and alignment loss optimization. Experiments demonstrate that ReAlnet significantly enhances the similarity between AI representations and human visual cortical responses, achieving superior performance over state-of-the-art vision models on fine-grained recognition and cross-modal generalization tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Despite advancements in artificial intelligence, object recognition models still lag behind in emulating visual information processing in human brains. Recent studies have highlighted the potential of using neural data to mimic brain processing; however, these often rely on invasive neural recordings from non-human subjects, leaving a critical gap in understanding human visual perception. Addressing this gap, we present, for the first time, ‘Re(presentational)Al(ignment)net’, a vision model aligned with human brain activity based on non-invasive EEG, demonstrating a significantly higher similarity to human brain representations. Our innovative image-to-brain multi-layer encoding framework advances human neural alignment by optimizing multiple model layers and enabling the model to efficiently learn and mimic human brain’s visual representational patterns across object categories and different modalities. Our findings suggest that ReAlnet represents a breakthrough in bridging the gap between artificial and human vision, and paving the way for more brain-like artificial intelligence systems.

Problem

• Aligning vision models with human EEG representations
• Bridging gap between artificial and human visual processing
• Developing non-invasive brain-like object recognition systems

Innovation

• Aligns vision model with human EEG data
• Uses multi-layer encoding framework for neural alignment
• Optimizes model layers to mimic brain representations