Federated Learning via Meta-Variational Dropout

2025-10-23

AI Summary

To address model overfitting and local model divergence caused by non-IID data in federated learning (FL), this paper proposes MetaVD—a personalized FL framework grounded in Bayesian meta-learning. Methodologically, MetaVD introduces a conditional variational Dropout posterior, wherein a shared hypernetwork dynamically generates client-specific Dropout rates, thereby unifying meta-posterior adaptation with federated posterior aggregation. This design simultaneously enables model personalization, parameter compression, and uncertainty calibration. Extensive experiments on multiple non-IID and sparse FL benchmarks demonstrate that MetaVD significantly improves classification accuracy and out-of-distribution generalization, yields better-calibrated uncertainty estimates, and reduces both communication overhead and overfitting risk compared to state-of-the-art baselines.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Metareasoning and Metaheuristics
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated Learning (FL) aims to train a global inference model from remotely distributed clients, gaining popularity due to its benefit of improving data privacy. However, traditional FL often faces challenges in practical applications, including model overfitting and divergent local models due to limited and non-IID data among clients. To address these issues, we introduce a novel Bayesian meta-learning approach called meta-variational dropout (MetaVD). MetaVD learns to predict client-dependent dropout rates via a shared hypernetwork, enabling effective model personalization of FL algorithms in limited non-IID data settings. We also emphasize the posterior adaptation view of meta-learning and the posterior aggregation view of Bayesian FL via the conditional dropout posterior. We conducted extensive experiments on various sparse and non-IID FL datasets. MetaVD demonstrated excellent classification accuracy and uncertainty calibration performance, especially for out-of-distribution (OOD) clients. MetaVD compresses the local model parameters needed for each client, mitigating model overfitting and reducing communication costs. Code is available at https://github.com/insujeon/MetaVD.

Problem

• Addressing model overfitting in federated learning with limited data
• Handling divergent local models from non-IID client data distributions
• Improving personalization and uncertainty calibration for OOD clients

Innovation

• Bayesian meta-learning with client-dependent dropout rates
• Hypernetwork predicts personalized dropout for non-IID data
• Compresses local parameters to reduce overfitting and communication