Connecting Federated ADMM to Bayes

2025-01-28

AI Summary

This work addresses the fundamental disconnect between optimization and probabilistic inference paradigms in federated learning. We establish, for the first time, a theoretical equivalence between federated ADMM and variational Bayes (VB): specifically, we prove that the ADMM dual variables exactly correspond to the parameters of Gaussian variational distributions at each client under the VB framework. Leveraging this insight, we propose two novel variational federated ADMM algorithms—supporting flexible covariance modeling and functional-space regularization—that seamlessly integrate deterministic optimization with probabilistic inference. Evaluated on multiple benchmark datasets, our methods achieve significantly faster convergence (average speedup of 2.1×) and higher model accuracy (up to +3.7%) compared to standard baselines. These results empirically validate the effectiveness and generalizability of cross-paradigm co-design in federated learning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

We provide new connections between two distinct federated learning approaches based on (i) ADMM and (ii) Variational Bayes (VB), and propose new variants by combining their complementary strengths. Specifically, we show that the dual variables in ADMM naturally emerge through the 'site' parameters used in VB with isotropic Gaussian covariances. Using this, we derive two versions of ADMM from VB that use flexible covariances and functional regularisation, respectively. Through numerical experiments, we validate the improvements obtained in performance. The work shows connection between two fields that are believed to be fundamentally different and combines them to improve federated learning.

Problem

• Federated Learning
• Bayesian Methods
• Alternating Direction Method of Multipliers

Innovation

• Federated Learning
• ADMM-VB Integration
• Performance Superiority