Online Dynamic Batching with Formal Guarantees for LLM Training

2026-06-18

AI Summary

This work addresses the inefficiency in large language model training caused by traditional offline batching, which suffers from padding waste and low GPU utilization due to unpredictable per-sample computational costs—such as post-tokenization sequence lengths and multimodal extensions. To overcome this, we propose Online Dynamic Batching (ODB), a system that dynamically forms batches at the DataLoader layer once true sample lengths become observable, while strictly preserving step alignment in Distributed Data Parallel (DDP) training. We formally define the Distributed Group Alignment Problem (DGAP) and devise a deadlock-free, bounded-termination scheduling mechanism. ODB is implemented as a plug-and-play solution requiring no model or kernel modifications, supporting both full fine-tuning and LoRA, as well as multimodal inputs. Experiments show ODB achieves 1.58–2.51× single-node and 1.71–3.78× two-node throughput gains over fixed-batch baselines on models like Qwen3-VL, with up to 4.43× speedup in production settings—all without compromising model quality.

Abstract

Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion. Unless one pays for a preprocessing- and augmentation-dependent length cache, batch construction is therefore blind to the quantity that determines padding, memory use, and GPU saturation. We introduce Online Dynamic Batching (ODB), a DataLoader-side drop-in system that moves batch formation to this point of accurate observability while preserving DDP step alignment. We formalize this synchronization requirement as the Distributed Group Alignment Problem and prove deadlock-free bounded termination with default join-mode identity coverage and opt-in non-join sample-quota closure. ODB requires no model, optimizer, or attention-kernel changes and is released as online-dynamic-batching with lightweight trainer adapters. Across public 2B/8B Qwen3-VL runs on UltraChat/LLaVA/ShareGPT4o, ODB improves literal emitted-sample throughput vs. fixed-batch Standard by 1.58-2.51x on single-node Full FT/LoRA and 1.71-3.78x on two-node Full FT, with Standard-comparable quality; production MM-Mix reaches 4.43x. Against GMT/BMT offline token-budget oracles, ODB is within 15% on UltraChat/LLaVA and faster on high-CV ShareGPT4o: 2.24-2.39x single-node Full FT/LoRA and 3.06-3.69x two-node Full FT. Together, ODB occupies the online/drop-in regime for high-heterogeneity LLM fine-tuning: large throughput gains at Standard-comparable quality, formal DGAP guarantees, and no length-cache precompute or kernel rewrites.

Problem

• Online Dynamic Batching
• LLM Training
• Batch Construction
• Sequence Length Heterogeneity
• Distributed Group Alignment

Innovation

• Online Dynamic Batching
• Distributed Group Alignment Problem
• LLM Training
• Throughput Optimization
• Drop-in DataLoader