HetCCL: Accelerating LLM Training with Heterogeneous GPUs

2026-01-30

AI Summary

This work addresses the lack of efficient collective communication support across heterogeneous GPUs from vendors such as NVIDIA and AMD in existing deep learning frameworks, which hinders large model training efficiency. To overcome this limitation, we propose HetCCL—the first library that enables high-performance cross-vendor GPU collective communication without requiring modifications to drivers or applications. HetCCL unifies NCCL and RCCL backends and constructs cross-vendor communication paths using RDMA, while introducing two novel mechanisms to ensure compatibility with mainstream deep learning frameworks. Experimental results demonstrate that HetCCL matches the performance of native communication libraries in homogeneous environments and significantly enhances scalability and training efficiency for large models in heterogeneous GPU clusters.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.

Problem

• heterogeneous GPUs
• collective communication
• LLM training
• multi-vendor GPU cluster

Innovation

• heterogeneous GPUs
• collective communication
• RDMA
• LLM training
• cross-vendor