Personalized Federated Learning with Residual Fisher Information for Medical Image Segmentation

2026-03-16

AI Summary

This work addresses the challenge of personalization in federated learning for medical image segmentation, where performance is hindered by heterogeneous client data distributions. The authors propose pFL-ResFIM, a novel framework that introduces a residual Fisher Information Matrix (FIM) to quantify the sensitivity of model parameters to domain shifts. By leveraging a spectral transfer strategy, the method estimates this matrix without accessing raw client data, enabling the separation of domain-sensitive and domain-invariant parameters. Only the domain-invariant components are aggregated across clients to construct personalized models. Enhanced by parameter-level adaptation and cross-domain style simulation, pFL-ResFIM significantly outperforms existing approaches on multiple public medical image datasets, demonstrating substantial improvements in federated personalized segmentation performance.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Federated learning enables multiple clients (institutions) to collaboratively train machine learning models without sharing their private data. To address the challenge of data heterogeneity across clients, personalized federated learning (pFL) aims to learn customized models for each client. In this work, we propose pFL-ResFIM, a novel pFL framework that achieves client-adaptive personalization at the parameter level. Specifically, we introduce a new metric, Residual Fisher Information Matrix (ResFIM), to quantify the sensitivity of model parameters to domain discrepancies. To estimate ResFIM for each client model under privacy constraints, we employ a spectral transfer strategy that generates simulated data reflecting the domain styles of different clients. Based on the estimated ResFIM, we partition model parameters into domain-sensitive and domain-invariant components. A personalized model for each client is then constructed by aggregating only the domain-invariant parameters on the server. Extensive experiments on public datasets demonstrate that pFL-ResFIM consistently outperforms state-of-the-art methods, validating its effectiveness.

Problem

• Personalized Federated Learning
• Medical Image Segmentation
• Data Heterogeneity
• Domain Discrepancy
• Privacy-Preserving

Innovation

• Personalized Federated Learning
• Residual Fisher Information Matrix
• Domain-Invariant Parameters
• Spectral Transfer
• Medical Image Segmentation