FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participation

2026-01-29

AI Summary

This work addresses the challenges of gradient noise, client drift, and participation bias arising from partial client participation in federated learning. To mitigate these issues under data heterogeneity, the authors propose FedAdaVR, an algorithm that integrates adaptive optimization with variance reduction by caching and reusing historical local updates to simulate the contributions of non-participating clients. Furthermore, they introduce a quantized variant, FedAdaVR-Quant, which reduces memory consumption by 50%–87.5% while preserving model performance. Theoretical analysis establishes, for the first time in the non-convex setting, the elimination of bias induced by partial participation. Empirical results demonstrate that the proposed method consistently outperforms existing approaches in both IID and non-IID scenarios.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Non-convex Optimization

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Federated learning (FL) encounters substantial challenges due to heterogeneity, leading to gradient noise, client drift, and partial client participation errors, the last of which is the most pervasive but remains insufficiently addressed in current literature. In this paper, we propose FedAdaVR, a novel FL algorithm aimed at solving heterogeneity issues caused by sporadic client participation by incorporating an adaptive optimiser with a variance reduction technique. This method takes advantage of the most recent stored updates from clients, even when they are absent from the current training round, thereby emulating their presence. Furthermore, we propose FedAdaVR-Quant, which stores client updates in quantised form, significantly reducing the memory requirements (by 50%, 75%, and 87.5%) of FedAdaVR while maintaining equivalent model performance. We analyse the convergence behaviour of FedAdaVR under general nonconvex conditions and prove that our proposed algorithm can eliminate partial client participation error. Extensive experiments conducted on multiple datasets, under both independent and identically distributed (IID) and non-IID settings, demonstrate that FedAdaVR consistently outperforms state-of-the-art baseline methods.

Problem

• federated learning
• client heterogeneity
• partial client participation
• gradient noise
• client drift

Innovation

• Federated Learning
• Variance Reduction
• Adaptive Optimization
• Client Participation
• Quantization