FedDuA: Doubly Adaptive Federated Learning

2025-05-16

AI Summary

To address the slow convergence of FedAvg in federated learning under non-IID data and anisotropic parameter spaces, this paper proposes a dual-adaptive optimization framework. The method jointly models cross-client statistical heterogeneity and coordinate-wise gradient heterogeneity at the server side, and introduces— for the first time—a dual adaptive step-size rule grounded in a mirror descent perspective for theoretical analysis. Without increasing communication rounds or client-side computational overhead, it achieves minimax-optimal step-size adaptation. Theoretically, it guarantees strict convergence for convex objectives. Empirically, it accelerates convergence by 30%–50% over FedAvg across diverse non-IID settings, while exhibiting robustness to hyperparameter selection.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federated learning, but it often suffers from slow convergence due to the heterogeneity of local datasets and anisotropy in the parameter space. In this work, we formalize the central server optimization procedure through the lens of mirror descent and propose a novel framework, called FedDuA, which adaptively selects the global learning rate based on both inter-client and coordinate-wise heterogeneity in the local updates. We prove that our proposed doubly adaptive step-size rule is minimax optimal and provide a convergence analysis for convex objectives. Although the proposed method does not require additional communication or computational cost on clients, extensive numerical experiments show that our proposed framework outperforms baselines in various settings and is robust to the choice of hyperparameters.

Problem

• Addresses slow convergence in federated learning due to data heterogeneity
• Proposes adaptive global learning rate selection for client and coordinate variations
• Ensures minimax optimality without extra communication or computational costs

Innovation

• Adaptive global learning rate selection
• Mirror descent optimization framework
• Minimax optimal step-size rule