A Queueing Theoretic Perspective on Low-Latency LLM Inference with Variable Token Length
Variable-length outputs in LLM interactive serving induce significant inference queuing latency due to output-token–dependent service times. Method: We propose a unified theoretical framework integrating M/G/1 and batch-service queueing models, the first to treat output token count as a stochastic service time. We jointly optimize the max-token limit and batch scheduling policies—fixed, dynamic, and elastic—to characterize their distinct latency behaviors under output-length uncertainty. Contribution/Results: Our analysis reveals the dominant impact of long-tail requests on mean queuing delay. Event-driven simulations validate model accuracy (<5% error): setting max-token = 256 reduces mean queuing delay by 38%; under load fluctuations, elastic batching cuts delay by 22% versus fixed batching. The core contribution is establishing a quantitative relationship between output-length variability and system latency, enabling principled co-optimization of inference parameters.