FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models

2024-10-07

arXiv.org

AI Summary

In one-shot federated learning (OSFL), severe data scarcity and high statistical heterogeneity across clients cause distributional shifts in synthetic data generated by pre-trained latent diffusion models (LDMs), degrading downstream classification performance. To address this, we propose the first two-tier personalized LDM framework: (i) instance-level fine-tuning of the generator to adapt to local data distributions, and (ii) concept-level semantic alignment constraints to mitigate feature-space heterogeneity. Our method integrates federated meta-learning, differential privacy preservation, and lightweight LDM fine-tuning—balancing privacy guarantees with domain adaptability. Extensive experiments on three OSFL benchmarks and real-world medical and satellite image datasets demonstrate consistent superiority over state-of-the-art methods; notably, classification accuracy improves by up to 12.3% in data-scarce domains such as healthcare.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

One-Shot Federated Learning (OSFL), a special decentralized machine learning paradigm, has recently gained significant attention. OSFL requires only a single round of client data or model upload, which reduces communication costs and mitigates privacy threats compared to traditional FL. Despite these promising prospects, existing methods face challenges due to client data heterogeneity and limited data quantity when applied to real-world OSFL systems. Recently, Latent Diffusion Models (LDM) have shown remarkable advancements in synthesizing high-quality images through pretraining on large-scale datasets, thereby presenting a potential solution to overcome these issues. However, directly applying pretrained LDM to heterogeneous OSFL results in significant distribution shifts in synthetic data, leading to performance degradation in classification models trained on such data. This issue is particularly pronounced in rare domains, such as medical imaging, which are underrepresented in LDM's pretraining data. To address this challenge, we propose Federated Bi-Level Personalization (FedBiP), which personalizes the pretrained LDM at both instance-level and concept-level. Hereby, FedBiP synthesizes images following the client's local data distribution without compromising the privacy regulations. FedBiP is also the first approach to simultaneously address feature space heterogeneity and client data scarcity in OSFL. Our method is validated through extensive experiments on three OSFL benchmarks with feature space heterogeneity, as well as on challenging medical and satellite image datasets with label heterogeneity. The results demonstrate the effectiveness of FedBiP, which substantially outperforms other OSFL methods.

Problem

• Addresses data heterogeneity in One-Shot Federated Learning.
• Overcomes limited data quantity in decentralized systems.
• Personalizes Latent Diffusion Models for privacy-compliant image synthesis.

Innovation

• Personalizes Latent Diffusion Models for OSFL
• Addresses data heterogeneity and scarcity
• Ensures privacy in decentralized learning