BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving

2026-07-09

AI Summary

This work addresses the inefficiency in diffusion-based large language model (dLLM) inference caused by divergent request convergence rates, which induce computational bubbles and tail latency, severely limiting batch throughput. To tackle this, the authors propose BlockServe, a novel framework featuring block-granularity continuous batching that instantly evicts completed requests at block boundaries. BlockServe integrates gather-scatter–based hybrid-state execution, dual-buffer expansion, parallel decoding, and computation-aware admission control to enable efficient scheduling and dynamic slot filling for heterogeneous requests. Evaluated on Dream and LLaDA models across five benchmarks, BlockServe achieves 1.9× to 10.6× higher throughput than Fast-dLLM while maintaining comparable generation quality.

Abstract

Efficient serving of diffusion large language models (dLLMs) is hindered by convergence heterogeneity: when batching multiple requests, different sequences converge at different rates, causing faster requests to stall behind slower stragglers and introducing compute bubbles and tail latency. We present BlockServe, a continuous batching framework that integrates block-grained scheduling -- immediately evicting completed requests at block boundaries -- with mixed-state execution that extends dual cache and parallel decoding to heterogeneous batches via gather-scatter indexing. Furthermore, a compute-aware admission controller expands effective batch capacity through token-budgeted refill. On Dream and LLaDA across five benchmarks, BlockServe achieves 1.9--10.6$\times$ throughput over Fast-dLLM with comparable generation quality, establishing block-grained scheduling as a foundation for high-throughput offline dLLM inference.

Problem

• diffusion LLM
• convergence heterogeneity
• continuous batching
• tail latency
• throughput

Innovation

• block-grained scheduling
• continuous batching
• mixed-state execution
• compute-aware admission control
• diffusion LLM serving