Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks

2024-08-26

arXiv.org

AI Summary

Heterogeneity across multimodal neuroimaging modalities—functional MRI (fMRI), diffusion tensor imaging (DTI), and structural MRI (sMRI)—hampers effective cross-modal integration for elucidating structure-function coordination underlying adolescent cognitive development. Method: We propose a brain-region–level alignment framework based on the Glasser parcellation and introduce a masked-weighted graph fusion mechanism to enable interpretable, connection-level multimodal integration. Departing from conventional unimodal, region-centric paradigms, we establish the first structural–functional coupling–driven analytical framework for cognitive development. Results: Evaluated on the Human Connectome Project Development (HCP-D) dataset, our method significantly improves prediction accuracy of cognitive behavioral measures, identifies discriminative anatomical features and core cross-modal connections, and systematically uncovers temporal developmental patterns wherein white-matter pathways guide functional connectivity maturation. This work provides a novel, multiscale paradigm for investigating human brain development.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal neuroimaging modeling has become a widely used approach but confronts considerable challenges due to heterogeneity, which encompasses variability in data types, scales, and formats across modalities. This variability necessitates the deployment of advanced computational methods to integrate and interpret these diverse datasets within a cohesive analytical framework. In our research, we amalgamate functional magnetic resonance imaging (fMRI), diffusion tensor imaging (DTI), and structural MRI (sMRI) into a cohesive framework. This integration capitalizes on the unique strengths of each modality and their inherent interconnections, aiming for a comprehensive understanding of the brain’s connectivity and anatomical characteristics. Utilizing the Glasser atlas for parcellation, we integrate imaging-derived features from various modalities—functional connectivity from fMRI, structural connectivity from DTI, and anatomical features from sMRI—within consistent regions. Our approach incorporates a masking strategy to differentially weight neural connections, thereby facilitating a holistic amalgamation of multimodal imaging data. This technique enhances interpretability at connectivity level, transcending traditional analyses centered on singular regional attributes. The model is applied to the Human Connectome Project’s Development study to elucidate the associations between multimodal imaging and cognitive functions throughout youth. The analysis demonstrates improved predictive accuracy and uncovers crucial anatomical features and essential neural connections, deepening our understanding of brain structure and function. This study not only advances multi-modal neuroimaging analytics by offering a novel method for the integrated analysis of diverse imaging modalities but also improves the understanding of intricate relationship between the brain’s structural and functional networks and cognitive development.

Problem

• Integrates fMRI, DTI, sMRI for brain connectivity analysis
• Addresses data heterogeneity in multimodal neuroimaging modeling
• Enhances interpretability of neural connections and cognitive functions

Innovation

• Integrates fMRI, DTI, sMRI via Graph Neural Networks
• Uses Glasser atlas for multimodal feature parcellation
• Applies masking strategy to weight neural connections