Boosting Asynchronous Decentralized Learning with Model Fragmentation

2024-10-16

arXiv.org

AI Summary

To address the slow convergence caused by stragglers in decentralized learning, this paper proposes DivShare, an asynchronous distributed algorithm where nodes partition model parameters into shards and concurrently transmit them to random neighbors via a sparse, randomized peer-to-peer topology—bypassing sequential full-model transfers. DivShare is the first decentralized learning algorithm with rigorous convergence guarantees under asynchronous communication delays. Its sharding mechanism jointly optimizes bandwidth efficiency and rapid participation of stragglers. On CIFAR-10, DivShare achieves a 3.9× speedup over AD-PSGD and improves test accuracy by 19.4%; on MovieLens, it reduces test loss by 9.5%. The core innovation lies in the principled integration of asynchronous updates, model sharding, and randomized sparse communication—delivering substantial gains in training efficiency and model performance for heterogeneous networks, all supported by theoretical convergence analysis.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: The sharing economy
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Decentralized learning (DL) is an emerging technique that allows nodes on the web to collaboratively train machine learning models without sharing raw data. Dealing with stragglers, i.e., nodes with slower compute or communication than others, is a key challenge in DL. We present DivShare, a novel asynchronous DL algorithm that achieves fast model convergence in the presence of communication stragglers. DivShare achieves this by having nodes fragment their models into parameter subsets and send, in parallel to computation, each subset to a random sample of other nodes instead of sequentially exchanging full models. The transfer of smaller fragments allows more efficient usage of the collective bandwidth and enables nodes with slow network links to quickly contribute with at least some of their model parameters. By theoretically proving the convergence of DivShare, we provide, to the best of our knowledge, the first formal proof of convergence for a DL algorithm that accounts for the effects of asynchronous communication with delays. We experimentally evaluate DivShare against two state-of-the-art DL baselines, AD-PSGD and Swift, and with two standard datasets, CIFAR-10 and MovieLens. We find that DivShare with communication stragglers lowers time-to-accuracy by up to 3.9x compared to AD-PSGD on the CIFAR-10 dataset. Compared to baselines, DivShare also achieves up to 19.4% better accuracy and 9.5% lower test loss on the CIFAR-10 and MovieLens datasets, respectively.

Problem

• Decentralized Learning
• Slow Network Environment
• Asynchronous Distributed Learning

Innovation

• DivShare
• Decentralized Learning
• Efficient Performance