Brain Foundation Models with Hypergraph Dynamic Adapter for Brain Disease Analysis

2025-05-01

AI Summary

To address the limitations of existing brain foundation models—including task specificity, poor generalization, and low clinical adaptation efficiency—this paper proposes SAM-Brain3D, a 3D foundation model, and Hypergraph Dynamic Adapter (HyDA). HyDA is the first to incorporate hypergraph structures into brain model adaptation, enabling patient-specific dynamic convolution kernel generation and joint multi-scale, multi-modal modeling—thereby breaking away from static adaptation paradigms. The method integrates a 3D Segment Anything Model (SAM) architecture, hypergraph neural networks, dynamic deformable convolutions, multi-modal feature fusion, and a lightweight adapter design. Evaluated across 14 MRI sub-modalities and six brain disorder tasks, it consistently surpasses state-of-the-art methods: segmentation Dice score improves by 3.2%, average classification accuracy increases by 2.8%, while requiring only 0.1M trainable adaptation parameters.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Computer Vision: 3D Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Brain diseases, such as Alzheimer's disease and brain tumors, present profound challenges due to their complexity and societal impact. Recent advancements in brain foundation models have shown significant promise in addressing a range of brain-related tasks. However, current brain foundation models are limited by task and data homogeneity, restricted generalization beyond segmentation or classification, and inefficient adaptation to diverse clinical tasks. In this work, we propose SAM-Brain3D, a brain-specific foundation model trained on over 66,000 brain image-label pairs across 14 MRI sub-modalities, and Hypergraph Dynamic Adapter (HyDA), a lightweight adapter for efficient and effective downstream adaptation. SAM-Brain3D captures detailed brain-specific anatomical and modality priors for segmenting diverse brain targets and broader downstream tasks. HyDA leverages hypergraphs to fuse complementary multi-modal data and dynamically generate patient-specific convolutional kernels for multi-scale feature fusion and personalized patient-wise adaptation. Together, our framework excels across a broad spectrum of brain disease segmentation and classification tasks. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art approaches, offering a new paradigm for brain disease analysis through multi-modal, multi-scale, and dynamic foundation modeling.

Problem

• Addressing task and data homogeneity in brain foundation models
• Enhancing generalization beyond segmentation or classification tasks
• Improving adaptation efficiency for diverse clinical brain disease tasks

Innovation

• SAM-Brain3D model trained on 66,000 brain image-label pairs
• Hypergraph Dynamic Adapter for multi-modal data fusion
• Dynamic patient-specific convolutional kernels for personalized adaptation