Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm

2026-06-21

AI Summary

This work addresses the challenges posed by heavy-tailed gradient and communication noise in federated learning over wireless IoT environments by proposing the VRA-FedSGD algorithm. It is the first method to jointly mitigate both types of heavy-tailed noise: momentum-based variance reduction combined with nonlinear mapping suppresses gradient noise, while a tailored variance-reduced aggregation mechanism alleviates the impact of communication noise. Theoretical analysis establishes convergence guarantees under both non-convex and strongly convex settings—achieving an expected convergence rate of 𝒪(K^{-(p-1)/(2p-1)}) in the non-convex case and an almost sure rate of 𝒪̃(K^{-(1-1/(p-ε))}) under strong convexity. Empirical results validate the algorithm’s effectiveness and robustness.

Abstract

Federated learning (FL) is an emerging distributed machine learning paradigm that enables local devices to jointly train a global model while keeping data decentralized and private. We propose a variance-reduction based algorithm, VRA-FedSGD, for FL in the presence of heavy-tailed gradient noise and communication noise, where these noises are prevalent in large-scale machine learning over wireless networks and Internet of Things deployments. VRA-FedSGD employs a momentum variance reduction technique together with a nonlinear mapping to mitigate heavy-tailed gradient noise, and uses a variance-reduced aggregation mechanism to suppress heavy-tailed communication noise. In the mean sense, VRA-FedSGD achieves a convergence rate of {\small$\mathcal{O}\left(K^{-(p-1)/(2p-1)}\right)$} for nonconvex objective functions, where $p$ is the tail index of heavy-tailed noise. In the almost sure sense, VRA-FedSGD achieves a convergence rate of $\tilde{\mathcal{O}}\left(K^{-(1-1/(p-ε))}\right)$ for strongly convex objective functions, where $ε$ is an arbitrarily small constant. Simulated experiments on a logistic regression problem with real-world data verify the effectiveness of VRA-FedSGD.

Problem

• federated learning
• heavy-tailed gradient noise
• communication noise
• variance reduction

Innovation

• federated learning
• heavy-tailed noise
• variance reduction
• nonlinear mapping
• convergence rate