Locally Coherent Parallel Decoding in Diffusion Language Models

2026-03-03

AI Summary

Discrete diffusion language models often suffer from grammatical errors and fragmented multi-word structures during parallel generation due to independent token sampling. To address this, this work proposes CoDiLA, the first approach that integrates a lightweight local autoregressive auxiliary model into the diffusion latent space. During parallel decoding, CoDiLA explicitly models intra-block sequential dependencies while preserving inter-block bidirectional context. The auxiliary model introduces only 0.6B additional parameters, maintaining sublinear latency while substantially improving generation coherence. On code generation tasks, CoDiLA establishes a new Pareto frontier by simultaneously advancing both accuracy and speed.

Abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel block generation while ensuring sequential validity within each block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.

Problem

• diffusion language models
• parallel decoding
• token coherence
• syntactic consistency
• joint dependencies

Innovation

• diffusion language models
• parallel decoding
• local autoregression
• coherence
• code generation