Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

2026-07-05

AI Summary

Diffusion language models (dLLMs) suffer from inefficient inference due to incompatibility between bidirectional attention mechanisms and autoregressive KV caching, which causes interference between prefill and decode phases and suboptimal resource allocation. This work proposes Sangam, the first system that holistically addresses this challenge by introducing a deficit token budget scheduler to dynamically coordinate computational resources. Sangam integrates a hybrid deployment architecture, KV cache reuse, and offloading of prefill overflow to decode nodes, enabling stall-free scheduling and high resource utilization. Evaluated on LLaDA-8B and Dream-7B models, Sangam reduces average latency by 9–20% and 8–20%, respectively, significantly outperforming existing approaches.

Abstract

Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others. Approximate caching techniques such as Fast-dLLM and dKV-Cache refresh KV activations repeatedly and reuse them across intervening decodes, inducing a repeated prefill/decode structure. This makes AR serving mechanisms relevant to dLLMs, but not directly applicable. dLLM decodes are block-sized rather than token-sized, prefills recur, and bidirectional attention precludes the chunked prefill mechanism used for stall-free colocated serving. We present Sangam, a serving system for cached dLLM inference. Sangam introduces a deficit token-budget scheduler that admits in-flight decodes first, admits whole indivisible prefills only when the accumulated token budget allows, and carries unused budget forward. This achieves amortized stall-free scheduling. Disaggregated serving avoids prefill-decode interference but suffers from prefill/decode resource partitioning problem. Sangam adopts a hybrid serving strategy, overflowing prefills onto decode workers to relieve prefill under-provisioning, and uses the same deficit-budget scheduler to protect those workers' decodes from the overflow. We show that like AR serving, dLLM serving design space is governed by prefill-decode interference and prefill/decode partitioning. Colocated serving is most effective on decode-heavy workloads, cutting mean latency by 9-20% over hybrid execution on LLaDA-8B ShareGPT; while hybrid execution is most effective on prefill-heavy workloads, cutting mean latency by 8-20% over colocated execution on Dream-7B arXiv. Sangam is available at https://github.com/UT-InfraAI/sangam.

Problem

• diffusion language models
• serving system
• prefill-decode interference
• resource partitioning
• KV caching

Innovation

• diffusion language models
• deficit token-budget scheduler
• hybrid serving
• KV caching
• prefill-decode interference