Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training

2026-06-15

AI Summary

This work addresses the high communication overhead in parallel training of large language models with long contexts under decentralized, low-bandwidth environments. The authors propose a dynamic subspace hybrid-based low-rank reparameterization method that leverages the intrinsic low-rank structure of activation outputs to enable highly efficient communication compression in context-parallel training. Without compromising convergence speed, the approach achieves over 95% communication compression, enabling successful training of billion-parameter models with context lengths exceeding 100,000 tokens on a 300 Mbps network. Remarkably, the resulting model performance matches that attained on a 100 Gbps high-speed cluster, demonstrating the method’s effectiveness in drastically reducing bandwidth requirements while maintaining scalability and training efficiency.

Abstract

Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95\% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.

Problem

• context parallelism
• communication efficiency
• decentralized training
• low-bandwidth networks
• large language models

Innovation

• mixture of subspaces
• context parallelism
• communication compression
• low-rank structure
• decentralized training