Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

2025-01-30

AI Summary

Communication bottlenecks severely hinder efficient distributed training of large language models (LLMs) across low-bandwidth networks. Method: This paper proposes a synergistic mechanism integrating streaming chunked parameter synchronization, computation-communication overlap, and INT8 gradient quantization. It introduces time-sliced streaming parameter updates and asynchronous pipelined scheduling, unified within the DiLoCo framework via chunked AllReduce and overlapped communication-computation. Contribution/Results: The approach eliminates reliance on high-bandwidth co-located hardware. For billion-parameter model training, it achieves convergence quality comparable to full-bandwidth baselines while significantly accelerating end-to-end training. Peak inter-worker communication bandwidth is reduced by two orders of magnitude (100×), with negligible impact on model quality.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Distributed Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.

Problem

• Large Language Models
• Reduced Network Communication
• Training Efficiency

Innovation

• Efficient Distributed Training
• Network Resource Optimization
• Communication-Computation Overlap