FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models

2026-06-22

AI Summary

Industrial-scale vision-language model (VLM) training suffers from significantly lower efficiency compared to unimodal large language models, primarily due to limitations in existing parallelization strategies and insufficient module decoupling. This work proposes a fully decoupled training paradigm that reformulates VLM training as a producer-consumer dataflow over a unified memory pool. By leveraging a global virtual address space, the visual encoder and language backbone can advance independently. The approach introduces a heterogeneous parallel allocator and a dynamic packing scheduler, uniquely integrating throughput matching into parallel strategy design and enabling runtime construction of micro-batches based on actual computational costs. Experiments demonstrate that this method achieves over 50% FLOPs utilization under real-world workloads and up to a 1.7× throughput improvement, substantially narrowing the efficiency gap with pure language model training.

Abstract

Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design that assigns uniform parallelism to heterogeneous modules or adopt a disaggregated deployment that separates modules while executing them as a batch-synchronized pipeline. In this paper, we highlight that the above solutions are still not sufficient, and VLM training can be further decoupled. To this end, we present FlowTrain, a flow-based decoupled training framework that reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool. The encoder and backbone can progress independently over a global virtual address space. Since this execution decoupling fundamentally changes the optimization objective of allocation and scheduling, FlowTrain further introduces a heterogeneous parallel allocator that assigns module-specific parallelism strategies by solving a throughput matching problem. The dynamic packing scheduler is used to construct balanced microbatches at runtime according to the actual LLM-side computation cost. Extensive experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement, narrowing the efficiency gap to LLM-only training.

Problem

• vision-language models
• distributed training
• training efficiency
• industrial-grade
• decoupled training

Innovation

• flow-based training
• decoupled execution
• heterogeneous parallelism
• dynamic microbatching
• vision-language models