Federated Fine-tuning of SAM-Med3D for MRI-based Dementia Classification

2025-08-29

AI Summary

This study addresses a critical design gap in deploying foundation models (FMs) for multi-center MRI-based dementia classification under federated learning (FL). We systematically investigate fine-tuning paradigms for SAM-Med3D, focusing on three core design dimensions: classifier head architecture, fine-tuning strategy (e.g., encoder freezing vs. full fine-tuning), and model aggregation method (FedAvg vs. advanced aggregation). Empirical evaluation is conducted on large-scale, multi-cohort brain MRI data. Results demonstrate that freezing the SAM-Med3D encoder and attaching a lightweight classifier head achieves performance comparable to full fine-tuning, while substantially reducing communication and computational overhead. Moreover, advanced aggregation methods consistently outperform FedAvg in generalization across clients. To our knowledge, this is the first work to establish an efficient, low-resource fine-tuning pathway for medical FMs in FL settings. It provides a reproducible, clinically viable deployment framework for FMs in distributed healthcare environments.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Other Foundations of Multi Agent Systems
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

While foundation models (FMs) offer strong potential for AI-based dementia diagnosis, their integration into federated learning (FL) systems remains underexplored. In this benchmarking study, we systematically evaluate the impact of key design choices: classification head architecture, fine-tuning strategy, and aggregation method, on the performance and efficiency of federated FM tuning using brain MRI data. Using a large multi-cohort dataset, we find that the architecture of the classification head substantially influences performance, freezing the FM encoder achieves comparable results to full fine-tuning, and advanced aggregation methods outperform standard federated averaging. Our results offer practical insights for deploying FMs in decentralized clinical settings and highlight trade-offs that should guide future method development.

Problem

• Evaluating federated learning for dementia diagnosis with foundation models
• Assessing impact of classification head architecture on MRI analysis
• Comparing fine-tuning strategies and aggregation methods in decentralized settings

Innovation

• Federated learning with SAM-Med3D foundation model
• Freezing encoder instead of full fine-tuning
• Advanced aggregation methods outperform standard averaging