Dynamic Chunking for Diffusion Language Models

2026-05-15

AI Summary

This work addresses the limitations of fixed-position chunking in discrete diffusion language models, which often disrupts semantic coherence and reduces modeling efficiency. The authors propose a dynamic semantic chunking mechanism that leverages differentiable Chunking Attention driven by learnable subspaces to cluster tokens into content-adaptive semantic blocks. Autoregressive diffusion denoising is then performed under block-wise causal masking. By replacing rigid positional chunks with semantically meaningful blocks, this approach enables end-to-end dynamic segmentation and strictly generalizes conventional block-based diffusion models. Experiments across model scales up to 1.5B parameters demonstrate that the method significantly outperforms both unstructured and position-based chunking baselines on downstream tasks, with performance gains emerging early in training and remaining consistent across model sizes.

Abstract

Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks, decoupling within-block parallel denoising from across-block conditioning. We argue that this rigid partition wastes structure already present in the sequence: blocks defined by position rather than by content separate semantically coherent tokens and group unrelated ones together. We introduce the \textbf{D}ynamic \textbf{C}hunking \textbf{D}iffusion \textbf{M}odel (DCDM), which replaces positional blocks with content-defined semantic chunks. At its core is Chunking Attention, a differentiable layer that routes tokens into $K$ clusters parameterized by learnable subspaces and shaped end-to-end by the diffusion objective. The resulting cluster assignments induce a chunk-causal attention mask under which a discrete diffusion denoiser factorizes the sequence likelihood autoregressively over semantic chunks, strictly generalizing block discrete diffusion. On downstream benchmarks at parameter scales up to 1.5B, DCDM consistently improves over both unstructured and positional-block diffusion baselines, with the advantage stable across scales and visible early in training.

Problem

• diffusion language models
• semantic chunking
• block discrete diffusion
• sequence modeling
• token clustering

Innovation

• Dynamic Chunking
• Diffusion Language Models
• Chunking Attention
• Semantic Clustering
• Block Discrete Diffusion