PipeWeaver: Addressing Data Dynamicity in Large Multimodal Model Training with Dynamic Interleaved Pipeline

2025-04-19

AI Summary

To address data dynamism and pipeline-stage imbalance arising from modality diversity in large multimodal model (LMM) training, this paper proposes a dynamic interleaved pipeline scheduling framework. Methodologically, it introduces: (1) a novel modality-aware dynamic batching adaptation mechanism enabling real-time, data-driven scheduling; (2) adaptive model partitioning coupled with hierarchical scheduling space search for memory-efficient optimization; and (3) integration of the SEMU stepwise simulator and spatiotemporal subgraph reuse techniques to support fine-grained, low-overhead optimization. Experimental evaluation demonstrates that the framework achieves up to 97.3% higher LMM training throughput compared to state-of-the-art systems, while significantly improving robustness and real-time adaptability to dynamic input data distributions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Large multimodal models (LMMs) have demonstrated excellent capabilities in both understanding and generation tasks with various modalities. While these models can accept flexible combinations of input data, their training efficiency suffers from two major issues: pipeline stage imbalance caused by heterogeneous model architectures, and training data dynamicity stemming from the diversity of multimodal data. In this paper, we present PipeWeaver, a dynamic pipeline scheduling framework designed for LMM training. The core of PipeWeaver is dynamic interleaved pipeline, which searches for pipeline schedules dynamically tailored to current training batches. PipeWeaver addresses issues of LMM training with two techniques: adaptive modality-aware partitioning and efficient pipeline schedule search within a hierarchical schedule space. Meanwhile, PipeWeaver utilizes SEMU (Step Emulator), a training simulator for multimodal models, for accurate performance estimations, accelerated by spatial-temporal subgraph reuse to improve search efficiency. Experiments show that PipeWeaver can enhance LMM training efficiency by up to 97.3% compared to state-of-the-art systems, and demonstrate excellent adaptivity to LMM training's data dynamicity.

Problem

• Addressing pipeline stage imbalance in LMM training
• Solving training data dynamicity from multimodal diversity
• Improving efficiency via dynamic interleaved pipeline scheduling

Innovation

• Dynamic interleaved pipeline scheduling
• Adaptive modality-aware partitioning
• Spatial-temporal subgraph reuse