FlashOverlap: Minimizing Tail Latency in Communication Overlap for Distributed LLM Training

2026-04-26

AI Summary

This work addresses the severe bottleneck imposed by communication tail latency in distributed large language model training. The authors propose a fine-grained communication-computation overlap method that, for the first time, enables precise elimination of tail latency. By decomposing collective communications—such as reduce-scatter and all-gather—into point-to-point operations and co-scheduling these with shard-level computation tasks, the approach maximizes overlap efficiency while remaining compatible with data parallelism and various tensor parallelism strategies, including TPSP and UP. Experimental results demonstrate that this method substantially reduces communication latency and significantly improves model FLOPS utilization (MFU) and throughput.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Distributed Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these parallelization strategies incur substantial data communication overhead significantly hindering computational efficiency. While communication-computation overlap presents a promising direction, existing data slicing based solutions suffer from tail latency. To overcome this limitation, this research introduces a novel communication-computation overlap technique to eliminate this tail latency in state of the art overlap methods for distributed LLM training. The aim of this technique is to effectively mitigate communication bottleneck of tensor parallelism and data parallelism for distributed training and inference. In particular, we propose a novel method termed Flash-Overlap that replaces conventional collective operations of reduce-scatter and all-gather with decomposed peer-to-peer (P2P) communication and schedules partitioned computations to enable fine-grained overlap. Our method provides an exact algorithm for reducing communication overhead that eliminates tail latency. Moreover, it presents a versatile solution compatible with data-parallel training and various tensor-level parallelism strategies, including TPSP and UP. Experimental evaluations demonstrate that our technique consistently achieves lower latency, superior Model FLOPS Utilization (MFU), and high throughput.

Problem

• tail latency
• communication overhead
• distributed LLM training
• communication-computation overlap
• tensor parallelism

Innovation

• communication-computation overlap
• tail latency
• peer-to-peer communication
• tensor parallelism
• distributed LLM training