FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning

2026-03-01

AI Summary

This work addresses the challenge of aligning brain functional connectivity networks (FCNs) with textual semantics, a limitation that hinders large language models (LLMs) from directly interpreting FCNs. To bridge this gap, the authors propose a novel graph-level multi-task instruction tuning paradigm: a multi-scale FCN encoder is constructed to project its features into the semantic space of an LLM, followed by multi-stage joint training guided by 19 diverse instructions encompassing demographic, phenotypic, and psychiatric attributes. This approach achieves, for the first time, end-to-end comprehension of resting-state fMRI–derived FCNs by an LLM. Evaluated on large-scale, multi-site datasets, the method demonstrates exceptional zero-shot generalization performance, significantly outperforming conventional supervised models and existing brain network foundation models, thereby enhancing flexibility and interpretability in neuroscientific applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.

Problem

• Functional Connectivity Network
• Large Language Model
• Multimodal Alignment
• Brain Imaging
• Resting-state fMRI

Innovation

• functional connectivity network
• large language model
• instruction tuning
• multi-task learning
• zero-shot generalization