BatchGen: An Architecture for Scalable and Efficient Batch Inference

2026-06-19

AI Summary

Existing inference engines struggle with extreme inter- and intra-sequence load imbalances in large-batch processing, resulting in low GPU utilization and reduced throughput. This work proposes a sequence coroutine computation model, representing each sequence as a fine-grained, event-driven coroutine that enables dynamic task reorganization at runtime. Introducing the sequence coroutine abstraction for the first time, the model achieves dynamic load balancing, expert-level large-batch scheduling, and cross-device task redistribution while maintaining high resource utilization even on memory-constrained devices. Built upon this model, the resulting cluster-scale batch inference system reduces end-to-end job completion time by up to 2.3× on a 128-GPU cluster and delivers up to 9.6× higher performance over the strongest offloading baseline on memory-limited accelerators.

Abstract

Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to $2.3\times$, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to $9.6\times$. We will open-source BatchGen at https://github.com/batchgen-project/batchgen

Problem

• batch inference
• load variation
• GPU utilization
• throughput loss
• scalable inference

Innovation

• sequence coroutine
• batch inference
• dynamic workload reorganization
• GPU utilization
• scalable inference