LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication

2025-11-12

AI Summary

To address the performance bottleneck caused by all-reduce communication in multi-node distributed inference of large language models (LLMs), this paper proposes NVRAR—a hierarchical all-reduce algorithm that integrates NVSHMEM with recursive doubling. NVRAR bypasses NCCL’s conventional cross-node communication path, enabling low-latency, high-bandwidth inter-node synchronization on HPE Slingshot and InfiniBand supercomputing infrastructures. Experimental results show that NVRAR reduces communication latency by 1.9×–3.6× over NCCL for message sizes ranging from 128 KB to 2 MB. For end-to-end batched inference of the Llama 3.1 405B model, it achieves up to a 1.72× reduction in latency. This work represents the first deep integration of NVSHMEM into all-reduce optimization for LLM inference, significantly improving strong scaling efficiency of multi-node tensor parallelism.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only across multiple GPUs but also across multiple nodes. In this work, we present a detailed performance study of multi-node distributed inference using LLMs on GPU-based supercomputers. We conduct experiments with several state-of-the-art inference engines alongside YALIS, a research-oriented prototype engine designed for controlled experimentation. We analyze the strong-scaling behavior of different model-parallel schemes and identify key bottlenecks. Since all-reduce operations are a common performance bottleneck, we develop NVRAR, a hierarchical all-reduce algorithm based on recursive doubling with NVSHMEM. NVRAR achieves up to 1.9x-3.6x lower latency than NCCL for message sizes between 128 KB and 2 MB on HPE Slingshot and InfiniBand interconnects. Integrated into YALIS, NVRAR achieves up to a 1.72x reduction in end-to-end batch latency for the Llama 3.1 405B model in multi-node decode-heavy workloads using tensor parallelism.

Problem

• Addressing communication bottlenecks in multi-node LLM distributed inference
• Developing fast hierarchical all-reduce algorithms for model parallelism
• Optimizing end-to-end latency for large language models across nodes

Innovation

• Hierarchical all-reduce algorithm using NVSHMEM
• Recursive doubling technique for communication optimization
• Multi-node tensor parallelism for distributed inference