MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding

2025-02-18

AI Summary

This work addresses three key bottlenecks in fMRI-to-text decoding: poor generalizability, task specificity, and limited performance. To this end, we propose Brain Instruction Tuning (BIT), a neuroscience-inspired paradigm. Methodologically, BIT introduces a brain-region-aware attention encoder that integrates spatiotemporal fMRI features with off-the-shelf large language models, coupled with a variable-input adaptation architecture and instruction-based fine-tuning—enabling, for the first time, subject-agnostic, instruction-driven universal decoding. Evaluated on a multi-task benchmark, BIT achieves a 12.0% improvement in downstream task performance, a 16.4% gain in cross-subject generalization, and a 25.0% enhancement in zero-shot task adaptation. Moreover, BIT generates interpretable brain-region attention maps, offering a novel framework for modeling neural language mechanisms.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model's ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by 12.0%, unseen subject generalization by 16.4%, and novel task adaptation by 25.0%. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.

Problem

• Decoding fMRI signals into text
• Improving subject-agnostic decoding performance
• Enhancing generalization across diverse tasks

Innovation

• Subject-agnostic fMRI-to-text decoding
• Neuroscience-informed attention mechanism
• Brain Instruction Tuning for versatile decoding