Personalized Federated Learning of Probabilistic Models: A PAC-Bayesian Approach

2024-01-16

arXiv.org

AI Summary

In federated learning, highly heterogeneous and scarce client data impede effective personalized modeling. Method: We propose PAC-PFL, a personalized federated learning algorithm grounded in the PAC-Bayesian framework that jointly learns a shared hyper-posterior distribution, enabling Bayesian personalized posterior inference per client. PAC-PFL is the first to integrate PAC-Bayesian generalization bounds with differential privacy to handle data-dependent priors, unifies global collaboration and local adaptation via hyper-posterior modeling, and combines variational Bayesian inference, federated hyperparameter learning, and Dirichlet-based non-IID data partitioning. Results: Experiments on photovoltaic power forecasting, FEMNIST, and Dirichlet-EMNIST demonstrate that PAC-PFL reduces average prediction error by 12.7% over baselines and improves expected calibration error (ECE) by over 40%, significantly enhancing generalization and predictive calibration—especially for clients with limited data.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated learning aims to infer a shared model from private and decentralized data stored locally by multiple clients. Personalized federated learning (PFL) goes one step further by adapting the global model to each client, enhancing the model's fit for different clients. A significant level of personalization is required for highly heterogeneous clients, but can be challenging to achieve especially when they have small datasets. To address this problem, we propose a PFL algorithm named PAC-PFL for learning probabilistic models within a PAC-Bayesian framework that utilizes differential privacy to handle data-dependent priors. Our algorithm collaboratively learns a shared hyper-posterior and regards each client's posterior inference as the personalization step. By establishing and minimizing a generalization bound on the average true risk of clients, PAC-PFL effectively combats over-fitting. PACPFL achieves accurate and well-calibrated predictions, supported by experiments on a dataset of photovoltaic panel power generation, FEMNIST dataset (Caldas et al., 2019), and Dirichlet-partitioned EMNIST dataset (Cohen et al., 2017).

Problem

• Enhances model fit for heterogeneous clients in federated learning
• Addresses overfitting in data-poor personalized federated learning scenarios
• Provides generalization bounds for new clients in probabilistic models

Innovation

• PAC-PFL infers shared hyper-posterior for personalization
• Minimizes PAC-Bayesian bound to prevent overfitting
• Provides generalization bounds for new clients