Brain-language fusion enables interactive neural readout and in-silico experimentation

2025-09-28

AI Summary

Current neural decoding methods predominantly rely on static, non-interactive paradigms, limiting open-ended natural language interaction with brain data. To address this, we propose a novel brain–language interface framework that directly maps fMRI neural activity into the latent space of large language models (LLMs), enabling end-to-end cross-modal alignment from neural representations to language. Our approach requires no task-specific fine-tuning and operates effectively in a zero-shot setting for image captioning and complex question answering. It further enables, for the first time, virtual cortical microstimulation–like experiments—marking a shift from passive neural recognition to generative, interactive decoding. Evaluated solely on neural data, our method significantly outperforms existing baselines in both generation quality and reasoning capability. These results validate the efficacy of latent-space fusion as a pathway toward interpretable and controllable brain–language interfaces.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) have revolutionized human-machine interaction, and have been extended by embedding diverse modalities such as images into a shared language space. Yet, neural decoding has remained constrained by static, non-interactive methods. We introduce CorText, a framework that integrates neural activity directly into the latent space of an LLM, enabling open-ended, natural language interaction with brain data. Trained on fMRI data recorded during viewing of natural scenes, CorText generates accurate image captions and can answer more detailed questions better than controls, while having access to neural data only. We showcase that CorText achieves zero-shot generalization beyond semantic categories seen during training. Furthermore, we present a counterfactual analysis that emulates in-silico cortical microstimulation. These advances mark a shift from passive decoding toward generative, flexible interfaces between brain activity and language.

Problem

• Integrates neural activity into LLM latent space
• Enables natural language interaction with brain data
• Achieves zero-shot generalization beyond training categories

Innovation

• Integrates neural activity into LLM latent space
• Enables natural language interaction with brain data
• Achieves zero-shot generalization beyond training categories