Heterogeneous Parallelism for Multimodal Large Language Model Training

2026-05-26

AI Summary

This work addresses the challenge of mismatched computational demands between vision encoders and large language models (LLMs) under long-context scenarios in multimodal large language model training, where conventional LLM-centric parallelization strategies fall short. The authors propose a heterogeneous parallel framework that decouples the encoder and LLM for the first time, enabling module-level independent parallelism and flexible device placement—either co-located or non-co-located. A boundary communicator ensures tensor semantic consistency across modules, while an extended scheduling mechanism supports diverse parallelism combinations, including tensor (TP), context (CP), pipeline (PP), data (DP), and expert (EP) parallelism. Experiments demonstrate that the proposed approach achieves up to 49.3% higher TFLOPS/GPU in co-located configurations and improves token throughput by 13.0% and TFLOPS/GPU by 9.6% in non-co-located settings, all while maintaining convergence performance on par with baseline methods.

Abstract

Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to inherit LLM-driven sharding and placement choices that can add communication, limit encoder parallelism, or constrain the LLM schedule; the mismatch is most pronounced at long contexts, where LLM context parallelism is needed for the fused multimodal sequence but encoder inputs remain bounded. We present heterogeneous parallelism for multimodal large language model training, an abstraction that lets modules in one end-to-end graph use independent layouts and rank placements, supporting colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. The key challenge is preserving boundary tensor semantics across independent layouts: forward activations must be materialized for the destination layout, while backward gradients must be routed back to the source layout. We address this with boundary communicators that implement forward and backward layout transforms, plus scheduling extensions for both placement modes. We evaluate optimized homogeneous, colocated heterogeneous, and non-colocated heterogeneous configurations across multimodal workloads and GPU scales to characterize when added layout and placement freedom exposes a better operating point. Across this sweep, colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. We validate loss convergence parity against homogeneous baselines and release the system as an open-source Megatron-LM extension.

Problem

• heterogeneous parallelism
• multimodal large language model
• training throughput
• layout mismatch
• long context

Innovation

• heterogeneous parallelism
• multimodal large language models
• boundary communicators
• layout transformation
• distributed training