Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo

2026-06-21

AI Summary

This work addresses the high communication overhead of DiLoCo in low-bandwidth environments and its sensitivity to stragglers and transient failures during large-scale distributed training. To mitigate these issues, the authors propose an approximate synchronization mechanism that, for the first time, integrates gossip-based hybrid communication into the DiLoCo framework. The synchronization process is factorized into two phases: a non-blocking phase that overlaps with computation to improve resource utilization, and a blocking phase that enhances inter-node consistency to ensure training stability. Evaluated on billion-parameter language model training, the proposed method achieves computational efficiency significantly higher than the original DiLoCo while maintaining comparable training progress and demonstrating markedly improved fault tolerance.

Abstract

To make large-scale distributed training practical outside high-bandwidth datacenters, we must reduce blocking, high-volume synchronization. While DiLoCo communicates infrequently, its outer synchronization remains bandwidth-heavy and brittle to stragglers and transient failures. We relax exact synchronization to approximate synchronization via mixing/gossip, which degrades gracefully under delays and communication failures. This allows us to factorize DiLoCo synchronization into a non-blocking mixing step that overlaps computation with no staleness, and a blocking mixing step that tightens worker agreement, yielding a tunable trade-off between compute utilization and optimization stability. On up to billion-parameter language models in low-bandwidth settings, our framework substantially improves compute utilization compared to DiLoCo, with training progress ranging from comparable to closely matching it, and is more robust to failures.

Problem

• distributed training
• synchronization
• low-bandwidth
• stragglers
• communication failures

Innovation

• gossip
• DiLoCo
• approximate synchronization
• non-blocking communication
• distributed training