Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

2026-09-13

AI Summary

研究解决去中心化GPU网络中大模型训练的通信瓶颈问题，通过异步双电路系统和频谱校正优化器提高压缩率和吞吐量。

Abstract

Decentralized training enables large-model training over low-end GPUs and internet-grade connections, but communication along both data-parallel and pipeline-parallel axes becomes the primary bottleneck. We study post-pretraining adaptation in this setting. We propose an asynchronous two-circuit system: a fast compressed training circuit drives throughput using activation masking for pipeline-parallel (PP) transfer and compressed data-parallel (DP) synchronization, while a slow anchor circuit runs occasional unmasked forward--backward passes off the critical path. Then, we introduce a spectral correction optimizer that uses these delayed anchor priors to denoise masked gradients without blocking the fast stream. Although prior work has found aggressive activation compression unreliable, we show that masking supports post-pretraining adaptation at high compression rates when anchored this way. Pipeline-parallel compression alone yields up to a $9\times$ throughput gain, and combining it with data-parallel compression increases beyond $40\times$ over internet-grade $\sim 200$Mbps connections, while matching dense uncompressed performance across domain adaptation and continual pretraining.

Problem

• decentralized training
• communication bottleneck
• large model adaptation
• pipeline-parallel
• data-parallel

Innovation

• asynchronous two-circuit system
• activation masking
• spectral correction optimizer
• high compression rates