BRACTIVE: A Brain Activation Approach to Human Visual Brain Learning

2024-05-29

arXiv.org

AI Summary

This study addresses the challenge of automatic region-of-interest (ROI) localization in visual cortex from multi-subject fMRI data—a problem where existing methods are limited to single-subject analysis and exhibit poor generalizability. We propose a Transformer-based cross-subject brain activation network that models interpretable, stimulus-response alignments between visual stimulus features (e.g., faces, bodies) and distributed fMRI responses across subjects, enabling joint identification and segmentation of category-specific visual ROIs. Our approach is the first to support end-to-end, multi-subject ROI localization; introduces a cross-modal, interpretable alignment mechanism grounded in neural encoding principles; and further refines vision models via neuroscientific feedback, yielding significant performance gains on downstream vision tasks. Extensive experiments validate both its neuroscientific plausibility—demonstrating anatomically and functionally coherent activations—and its strong cross-task generalization capability.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The human brain is a highly efficient processing unit, and understanding how it works can inspire new algorithms and architectures in machine learning. In this work, we introduce a novel framework named Brain Activation Network (BRACTIVE), a transformer-based approach to studying the human visual brain. The main objective of BRACTIVE is to align the visual features of subjects with corresponding brain representations via fMRI signals. It allows us to identify the brain's Regions of Interest (ROI) of the subjects. Unlike previous brain research methods, which can only identify ROIs for one subject at a time and are limited by the number of subjects, BRACTIVE automatically extends this identification to multiple subjects and ROIs. Our experiments demonstrate that BRACTIVE effectively identifies person-specific regions of interest, such as face and body-selective areas, aligning with neuroscience findings and indicating potential applicability to various object categories. More importantly, we found that leveraging human visual brain activity to guide deep neural networks enhances performance across various benchmarks. It encourages the potential of BRACTIVE in both neuroscience and machine intelligence studies.

Problem

• Aligns visual features with brain fMRI representations
• Identifies brain Regions of Interest across multiple subjects
• Enhances neural network performance using brain activity guidance

Innovation

• Transformer-based framework for brain activation study
• Aligns visual features with fMRI brain representations
• Automatically identifies multiple subjects' ROIs simultaneously