Optimus: Elastic Decoding for Efficient Diffusion LLM Serving

2026-05-23

AI Summary

Diffusion-based large language models struggle to balance GPU utilization and computational efficiency under dynamic workloads due to fixed decoding block sizes. This work proposes an elastic decoding mechanism that, for the first time, treats decoding granularity as a runtime-controllable variable. By integrating chunked decoding with saturation-aware closed-loop scheduling, the approach enables fine-grained elastic inference without requiring model retraining. Through system-level optimizations and a customized attention kernel, the method achieves up to 6.1× higher throughput compared to autoregressive decoding and 4.3× improvement over fixed-block diffusion decoding, while maintaining low latency and performance stability across varying workloads.

Abstract

Large language model (LLM) serving is fundamentally limited by inefficient hardware utilization. Autoregressive (AR) decoding underutilizes GPUs due to its strictly sequential execution, while diffusion LLMs (DLLMs) improve throughput by decoding multiple tokens per iteration. However, fixed block-size diffusion decoding exhibits strong load sensitivity: large blocks exploit idle GPU resources under low load, but saturate early and incur substantial redundant computation under high load. As a result, throughput gains vanish beyond saturation, and no single decoding granularity performs well across dynamic serving workloads. We present Optimus, a serving system that enables elastic decoding for diffusion LLMs by dynamically adapting decoding granularity to runtime load. The key idea is to treat decoding granularity as a runtime control variable, balancing GPU utilization and token efficiency. Optimus combines chunked decoding, which enables fine-grained execution without retraining, with saturation-aware scheduling, a closed-loop mechanism that selects chunk sizes based on runtime conditions. Together with system-level optimizations and customized attention kernels, Optimus achieves significant performance improvements while preserving model accuracy. Experiments show that Optimus delivers up to 6.1x throughput improvement over AR decoding and 4.3x improvement over fixed-block diffusion LLM, while maintaining stable performance across diverse load regimes and improving end-to-end serving capacity under latency constraints. The source code is available at https://github.com/dubcyfor3/Optimus.

Problem

• diffusion LLM
• elastic decoding
• GPU utilization
• decoding granularity
• serving efficiency

Innovation

• elastic decoding
• diffusion LLM
• chunked decoding
• saturation-aware scheduling
• GPU utilization