Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation

2025-02-26

AI Summary

Traditional psychological experiments rely on manual annotation of naturalistic stimuli, resulting in low ecological validity and limited capacity to characterize the neural representation of visual semantics. Method: This study introduces a novel paradigm leveraging multimodal large language models (MLLMs) as semantic proxies: visual question answering (VQA) automatically extracts image semantics, integrated with fMRI-based neural decoding and functional brain network analysis to construct the first whole-brain functional network grounded in LLM-derived semantics. Contribution/Results: LLM-derived semantic representations significantly predict category-specific fMRI responses (e.g., faces, buildings); identified brain semantic clusters align with established cognitive functional divisions and contextual associations, revealing a laminar hierarchical organization of semantic processing across cortical regions. This approach overcomes the bottleneck of manual annotation, substantially enhancing both ecological validity and scalability in studying neural mechanisms underlying naturalistic vision.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Traditional psychological experiments utilizing naturalistic stimuli face challenges in manual annotation and ecological validity. To address this, we introduce a novel paradigm leveraging multimodal large language models (LLMs) as proxies to extract rich semantic information from naturalistic images through a Visual Question Answering (VQA) strategy for analyzing human visual semantic representation. LLM-derived representations successfully predict established neural activity patterns measured by fMRI (e.g., faces, buildings), validating its feasibility and revealing hierarchical semantic organization across cortical regions. A brain semantic network constructed from LLM-derived representations identifies meaningful clusters reflecting functional and contextual associations. This innovative methodology offers a powerful solution for investigating brain semantic organization with naturalistic stimuli, overcoming limitations of traditional annotation methods and paving the way for more ecologically valid explorations of human cognition.

Problem

• Model brain semantic representation
• Overcome manual annotation challenges
• Validate LLM-derived neural predictions

Innovation

• Large Language Models
• Visual Question Answering
• fMRI Neural Patterns