Distributed Event-Based Learning via ADMM

2024-05-17

arXiv.org

AI Summary

To address high communication overhead, strong data heterogeneity, and weak fault tolerance in distributed learning, this paper proposes an event-triggered distributed ADMM algorithm (ET-ADMM). ET-ADMM is the first to tightly integrate event-triggered communication with ADMM, guaranteeing convergence without assuming data homogeneity and enabling communication reduction via transmission omission. It explicitly models communication failures to enhance system robustness. Theoretical analysis establishes convergence for both convex and nonconvex objectives under arbitrary data heterogeneity. Experiments on MNIST and CIFAR-10 demonstrate over 35% reduction in total communication volume compared to FedAvg, FedProx, SCAFFOLD, and FedADMM, while achieving superior robustness to statistical heterogeneity and accelerated convergence.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

We consider a distributed learning problem, where agents minimize a global objective function by exchanging information over a network. Our approach has two distinct features: (i) It substantially reduces communication by triggering communication only when necessary, and (ii) it is agnostic to the data-distribution among the different agents. We therefore guarantee convergence even if the local data-distributions of the agents are arbitrarily distinct. We analyze the convergence rate of the algorithm both in convex and nonconvex settings and derive accelerated convergence rates for the convex case. We also characterize the effect of communication failures and demonstrate that our algorithm is robust to these. The article concludes by presenting numerical results from distributed learning tasks on the MNIST and CIFAR-10 datasets. The experiments underline communication savings of 35% or more due to the event-based communication strategy, show resilience towards heterogeneous data-distributions, and highlight that our approach outperforms common baselines such as FedAvg, FedProx, SCAFFOLD and FedADMM.

Problem

• Distributed learning via ADMM
• Reduces communication with event-triggering
• Robust to heterogeneous data-distributions

Innovation

• Event-triggered communication reduces network traffic.
• Algorithm converges despite diverse data distributions.
• Robust to communication failures with accelerated convergence.