Distributed Quasi-Newton Method for Fair and Fast Federated Learning

2025-01-18

AI Summary

In federated learning, quasi-Newton methods often suffer from slow convergence and unfair model performance across clients due to statistical heterogeneity of local data. To address this, we propose DQN-Fed—the first framework that jointly integrates distributed quasi-Newton optimization with explicit fairness constraints. It enforces coordinated local loss descent, adapts Hessian approximations per client, and performs fairness-aware global aggregation, thereby synchronously optimizing all clients’ model performance each round. We theoretically establish its linear-to-quadratic convergence rate, breaking the conventional trade-off between fairness and efficiency. Empirical evaluation across diverse federated datasets shows that DQN-Fed achieves, on average, a 3.2% higher accuracy than state-of-the-art fair FL baselines, reduces inter-client performance variance by 37%, and cuts convergence iterations by 41%.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

Federated learning (FL) is a promising technology that enables edge devices/clients to collaboratively and iteratively train a machine learning model under the coordination of a central server. The most common approach to FL is first-order methods, where clients send their local gradients to the server in each iteration. However, these methods often suffer from slow convergence rates. As a remedy, second-order methods, such as quasi-Newton, can be employed in FL to accelerate its convergence. Unfortunately, similarly to the first-order FL methods, the application of second-order methods in FL can lead to unfair models, achieving high average accuracy while performing poorly on certain clients' local datasets. To tackle this issue, in this paper we introduce a novel second-order FL framework, dubbed extbf{d}istributed extbf{q}uasi- extbf{N}ewton extbf{fed}erated learning (DQN-Fed). This approach seeks to ensure fairness while leveraging the fast convergence properties of quasi-Newton methods in the FL context. Specifically, DQN-Fed helps the server update the global model in such a way that (i) all local loss functions decrease to promote fairness, and (ii) the rate of change in local loss functions aligns with that of the quasi-Newton method. We prove the convergence of DQN-Fed and demonstrate its extit{linear-quadratic} convergence rate. Moreover, we validate the efficacy of DQN-Fed across a range of federated datasets, showing that it surpasses state-of-the-art fair FL methods in fairness, average accuracy and convergence speed.

Problem

• Federated Learning
• Fairness
• Quasi-Newton Methods

Innovation

• DQN-Fed
• Distributed Quasi-Newton Method
• Federated Learning Fairness