HyperParallel-Mpipe: A Composable Algebra System for Optimizing MLLM Training over Supernode Clusters

2026-07-03

AI Summary

This work addresses the low computational resource utilization and significantly lower Model FLOPs Utilization (MFU) observed in large-scale multimodal large language model (MLLM) training compared to text-only LLMs. To overcome these limitations, the authors propose Mpipe, a novel system that introduces, for the first time, a composable scheduling algebra enabling multimodal-aware heterogeneous parallel scheduling. Mpipe further incorporates a transpose-based dynamic remapping strategy that flexibly schedules modality encoder computations into pipeline idle regions, thereby transcending the constraints of conventional static pipelining. Combined with hyper-node cluster optimizations, Mpipe achieves a 2.70× speedup in small-scale training and a 1.21× speedup on 512 Ascend 910C accelerators for large-scale MLLM training.

Abstract

Modern AI applications have expanded beyond text-only interaction into a wide range of multimodal scenarios, making multimodal large language models (MLLMs) crucial for both research and industry. However, compared with traditional decoder-only LLM training, large-scale MLLM training often shows much lower MFU. We analyze the key pain points in MLLM training and introduce Mpipe, which uses a schedule algebra to derive concrete runtime behavior from a compact schedule specification. From this algebra, Mpipe derives transpose, a multimodal-aware heterogeneous parallel schedule that remaps modality-encoder computation into otherwise idle pipeline regions. On Ascend 910C NPU clusters, Mpipe achieves 2.70x speedup in a small-scale setting and 1.21x speedup in a 512-card large-scale setting.

Problem

• Multimodal Large Language Models
• Training Efficiency
• Model FLOPs Utilization
• Distributed Training
• Heterogeneous Parallelism

Innovation

• schedule algebra
• multimodal-aware parallelism
• pipeline optimization
• heterogeneous scheduling
• MLLM training