Multimodal Deep Learning for Dynamic and Static Neuroimaging: Integrating MRI and fMRI for Alzheimer Disease Analysis

2026-03-09

AI Summary

This study addresses the insufficient integration of structural and functional neuroimaging data in multi-class classification across the Alzheimer’s disease spectrum—including Alzheimer’s disease, mild cognitive impairment, and normal cognition—by proposing a multimodal deep learning framework that jointly models 3D MRI structural features and dynamic fMRI temporal activity. The framework employs a 3D CNN to extract spatial structural information and an LSTM to capture functional temporal dependencies, enabling spatiotemporal joint learning through feature fusion. Evaluated on a small paired dataset of only 29 subjects, the multimodal model—after targeted data augmentation—demonstrates significantly improved classification stability and generalization, whereas single-modality models show limited gains even with large-scale augmentation. This work provides the first empirical validation of multimodal fusion efficacy in small-sample settings and reveals a critical interaction between modality synergy and data scale.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Magnetic Resonance Imaging (MRI) provides detailed structural information, while functional MRI (fMRI) captures temporal brain activity. In this work, we present a multimodal deep learning framework that integrates MRI and fMRI for multi-class classification of Alzheimer Disease (AD), Mild Cognitive Impairment, and Normal Cognitive State. Structural features are extracted from MRI using 3D convolutional neural networks, while temporal features are learned from fMRI sequences using recurrent architectures. These representations are fused to enable joint spatial-temporal learning. Experiments were conducted on a small paired MRI-fMRI dataset (29 subjects), both with and without data augmentation. Results show that data augmentation substantially improves classification stability and generalization, particularly for the multimodal 3DCNN-LSTM model. In contrast, augmentation was found to be ineffective for a large-scale single-modality MRI dataset. These findings highlight the importance of dataset size and modality when designing augmentation strategies for neuroimaging-based AD classification.

Problem

• Alzheimer Disease
• Multimodal Neuroimaging
• MRI
• fMRI
• Deep Learning

Innovation

• multimodal deep learning
• 3DCNN-LSTM
• neuroimaging fusion
• data augmentation
• Alzheimer's disease classification