Tangram: Hiding GPU Heterogeneity for Efficient LLM Parallelization

2026-06-15

AI Summary

This work addresses the inefficiency of existing automatic parallelization methods for large language models (LLMs) on heterogeneous GPU clusters, where the combinatorial explosion of the search space hinders joint optimization of parallel strategies and memory usage. The authors propose a novel approach that decouples parallel planning from hardware heterogeneity by identifying homogeneous GPU “islands” and presenting a unified view to off-the-shelf parallelizers such as Metis or Sailor. By integrating model slicing with load-balanced pipeline scheduling, the method efficiently reuses existing parallelization strategies without modifying their internal logic. A narrow API enables seamless scaling to large clusters, while a pruning mechanism filters infeasible plans. Experiments demonstrate up to 2.3× higher training throughput compared to state-of-the-art heterogeneous parallelizers, significantly improving scalability.

Abstract

The scale of LLM training jobs requires parallelization planning over large GPU clusters. Due to different GPU types and interconnects added over time, these GPU clusters are increasingly heterogeneous. Automatic LLM parallelizers can search for parallelization plans but face an exploding search space with heterogeneous GPUs. To make search tractable in heterogeneous GPU clusters, parallelizers often omit types of parallelism (e.g., expert parallelism) or memory-saving techniques (e.g., ZeRO), which results in worse plans. We describe Tangram, a system that enables the use of existing heterogeneity-unaware LLM parallelizers in heterogeneous GPU clusters by decoupling parallelization planning from GPU heterogeneity. For this, Tangram exploits two insights: (1) since bulk purchases result in sets of GPUs with similar compute, memory, and connectivity, Tangram can expose such homogeneous GPU islands to existing parallelizers; and (2) parallelizers commonly first partition models and then parallelize partitions. Tangram can compose such model slices, assigned to GPU islands, into work-balanced pipelines for high throughput. Tangram integrates with existing parallelizers through a narrow API, which relies on the enumeration of model-slice/island pairs. Tangram achieves up to 2.3x higher training throughput than current heterogeneous parallelizers (Metis and Sailor) and scales to large GPU clusters by pruning enumerated plans.

Problem

• LLM parallelization
• GPU heterogeneity
• parallelization planning
• heterogeneous clusters
• search space explosion

Innovation

• heterogeneous GPU clusters
• LLM parallelization
• homogeneous GPU islands
• model slicing
• pipeline composition