pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation

2025-09-19

AI Summary

To address the dual challenges of privacy preservation and data heterogeneity in medical image segmentation, this paper proposes the first Segment Anything Model (SAM)-based personalized federated learning framework. To overcome SAM’s prohibitively large encoder—which impedes efficient adaptation to federated settings—we design a decoupled global–local fine-tuning mechanism. Specifically, we introduce a Localized Mixture-of-Experts (L-MoE) module to preserve institution-specific feature representations and integrate knowledge distillation to mitigate over-generalization. Crucially, our framework enables joint optimization of global knowledge aggregation and local representation personalization—without sharing raw patient data. Evaluated on two public medical imaging benchmarks, the method achieves significant segmentation accuracy gains (average Dice score improvement of 3.2%) and demonstrates strong cross-domain generalizability, while reducing communication overhead by approximately 37%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Medical image segmentation is crucial for computer-aided diagnosis, yet privacy constraints hinder data sharing across institutions. Federated learning addresses this limitation, but existing approaches often rely on lightweight architectures that struggle with complex, heterogeneous data. Recently, the Segment Anything Model (SAM) has shown outstanding segmentation capabilities; however, its massive encoder poses significant challenges in federated settings. In this work, we present the first personalized federated SAM framework tailored for heterogeneous data scenarios in medical image segmentation. Our framework integrates two key innovations: (1) a personalized strategy that aggregates only the global parameters to capture cross-client commonalities while retaining the designed L-MoE (Localized Mixture-of-Experts) component to preserve domain-specific features; and (2) a decoupled global-local fine-tuning mechanism that leverages a teacher-student paradigm via knowledge distillation to bridge the gap between the global shared model and the personalized local models, thereby mitigating overgeneralization. Extensive experiments on two public datasets validate that our approach significantly improves segmentation performance, achieves robust cross-domain adaptation, and reduces communication overhead.

Problem

• Personalizing federated learning for medical image segmentation
• Addressing SAM model challenges in federated heterogeneous settings
• Reducing communication overhead while improving segmentation performance

Innovation

• Personalized federated SAM framework integration
• Decoupled global-local fine-tuning mechanism
• Teacher-student knowledge distillation paradigm