Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning

2024-02-19

arXiv.org

AI Summary

Statistical heterogeneity across clients in federated learning severely limits the performance of unsupervised tasks—such as dimensionality reduction and diffusion modeling—due to divergent local data distributions. Method: This work proposes the first personalized unsupervised federated learning framework, built upon a hierarchical Bayesian model that adaptively balances client-specific characteristics with global collaboration. We design personalized PCA and personalized denoising diffusion probabilistic models (DDPMs), and establish the first theoretical framework proving that collaborative learning under heterogeneity can yield sample amplification gains. Contribution/Results: Our theoretical analysis characterizes performance dependence on both heterogeneity level and local sample size. Extensive experiments on synthetic and real-world datasets demonstrate significant improvements over non-personalized and state-of-the-art baseline methods in reconstruction fidelity (for dimensionality reduction) and generation quality (for diffusion modeling), while ensuring convergence guarantees.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Statistical heterogeneity of clients' local data is an important characteristic in federated learning, motivating personalized algorithms tailored to the local data statistics. Though there has been a plethora of algorithms proposed for personalized supervised learning, discovering the structure of local data through personalized unsupervised learning is less explored. We initiate a systematic study of such personalized unsupervised learning by developing algorithms based on optimization criteria inspired by a hierarchical Bayesian statistical framework. We develop adaptive algorithms that discover the balance between using limited local data and collaborative information. We do this in the context of two unsupervised learning tasks: personalized dimensionality reduction and personalized diffusion models. We develop convergence analyses for our adaptive algorithms which illustrate the dependence on problem parameters (e.g., heterogeneity, local sample size). We also develop a theoretical framework for personalized diffusion models, which shows the benefits of collaboration even under heterogeneity. We finally evaluate our proposed algorithms using synthetic and real data, demonstrating the effective sample amplification for personalized tasks, induced through collaboration, despite data heterogeneity.

Problem

• Federated Learning
• Unsupervised Environment
• Data Heterogeneity

Innovation

• Hierarchical Bayesian Method
• Federated Learning
• Unsupervised Personalization