Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

2026-06-26

AI Summary

Discrete diffusion language models face an architectural tension in parallel generation: bidirectional attention is incompatible with KV caching, while causal attention lacks access to rightward context. This work proposes the Recurrent-Reverse Language Model (R2LM), which innovatively combines causal attention with a lightweight reverse Mamba state-space model to establish an asymmetric bidirectional context mechanism. This design preserves the efficiency of KV caching while incorporating compressed right-side contextual information. Large-scale continued pretraining based on Qwen3-1.7B demonstrates that R2LM achieves 2.4–12.9× speedup over bidirectional dLLMs and outperforms autoregressive baselines by 1.9–2.9× in batch serving scenarios. Moreover, it surpasses causal models on most benchmarks and delivers superior average performance compared to existing bidirectional dLLMs.

Abstract

Discrete diffusion language models (dLLMs) recover masked tokens in parallel, offering significant speedups over autoregressive (AR) generation. However, such promising frameworks face a fundamental architectural design dilemma: \ding{182} Adopting bidirectional attention achieves strong generation quality by allowing each position to access the full context, but is inherently incompatible with KV caching, limiting inference throughput in batch-serving scenarios; \ding{183} Conversely, causal attention enables efficient cached inference but loses all right-side context, substantially degrading generation quality. This paper introduces Bifocal dLLMs, a new paradigm that resolves this dilemma through \emph{asymmetric bidirectional context}. Analogous to bifocal lenses, we instantiate the paradigm as \textbf{R2LM} (Right-to-Left Mamba), which combines two complementary mechanisms: $a$) standard causal attention providing precise left-context with full KV cache compatibility, while $b$) a lightweight reverse Mamba SSM sidecar supplying compressed right-side context without breaking cacheability. Comprehensive experiments on continued pretraining of Qwen3-1.7B with 60B tokens demonstrate that R2LM achieves $2.4\times$ to $12.9\times$ higher throughput than bidirectional dLLMs and $1.9\times$ to $2.9\times$ speedup over AR baselines in batch serving through parallel decoding with KV caching, while exceeding the causal baseline on most benchmarks and surpassing the bidirectional dLLM on average.

Problem

• discrete diffusion language models
• bidirectional attention
• causal attention
• KV caching
• parallel generation

Innovation

• bifocal diffusion
• asymmetric bidirectional context
• KV caching
• parallel generation
• Mamba SSM