FedCRL: Personalized Federated Learning with Contrastive Shared Representations for Label Heterogeneity in Non-IID Data

2024-04-27

arXiv.org

AI Summary

To address accuracy degradation and fairness deterioration in federated learning under non-IID data—particularly due to label distribution skew and client-level data scarcity—this paper proposes FedCRL, a personalized federated learning framework. FedCRL introduces a contrastive shared representation mechanism that enables clients to collaboratively learn prototypical semantic representations. It further combines shallow-layer parameter sharing with an adaptive local weighted aggregation strategy to enhance participation fairness for data-scarce clients while preserving privacy. Technically, the framework integrates contrastive learning, hierarchical parameter sharing, and federated optimization. Extensive experiments on multiple non-IID benchmark datasets demonstrate that FedCRL significantly outperforms state-of-the-art federated methods: it achieves new SOTA performance in both overall accuracy and fairness metrics—e.g., improving the worst-client accuracy by up to 12.3%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Bias, Fairness & Privacy
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Heterogeneity resulting from label distribution skew and data scarcity can lead to inaccuracy and unfairness in intelligent communication applications that mainly rely on distributed computing. To deal with it, this paper proposes a novel personalized federated learning algorithm, named Federated Contrastive Shareable Representations (FedCoSR), to facilitate knowledge sharing among clients while maintaining data privacy. Specifically, parameters of local models' shallow layers and typical local representations are both considered shareable information for the server and aggregated globally. To address poor performance caused by label distribution skew among clients, contrastive learning is adopted between local and global representations to enrich local knowledge. Additionally, to ensure fairness for clients with scarce data, FedCoSR introduces adaptive local aggregation to coordinate the global model involvement in each client. Our simulations demonstrate FedCoSR's effectiveness in mitigating label heterogeneity by achieving accuracy and fairness improvements over existing methods on datasets with varying degrees of label heterogeneity.

Problem

• Addresses label distribution skew in federated learning
• Mitigates performance degradation from non-IID data heterogeneity
• Ensures fairness for clients with scarce data resources

Innovation

• Federated learning with shareable representations
• Contrastive learning for label distribution skew
• Adaptive local aggregation for data scarcity