FLARE: Diffusion for Hybrid Language Model

2026-06-01

AI Summary

This work addresses the challenge of deploying large language models with low latency, as autoregressive models suffer from sequential decoding bottlenecks, while existing diffusion-based language models struggle to maintain performance and training efficiency under hybrid attention architectures. The authors propose FLARE, a novel framework that systematically converts hybrid-attention autoregressive models into parallelizable diffusion models for the first time. By introducing a token-equal joint training objective, hardware-aware operators, and a unified inference mechanism, FLARE enables both speculative autoregressive and parallel diffusion generation from a single checkpoint. Experiments demonstrate that, with limited post-training data, FLARE matches state-of-the-art open-source diffusion models across multiple scales and significantly improves throughput in single-GPU concurrent serving, outperforming current baselines.

Abstract

Autoregressive (AR) large language models (LLMs) have achieved broad practical success, but sequential decoding remains a key bottleneck for low-latency deployment. Recent efficient-inference work has progressed along two axes: reducing the cost of each model invocation through efficient architectures, and reducing serial decoding steps through parallel generation. Hybrid attention backbones address the former, while diffusion language models (dLLMs) pursue the latter via iterative parallel denoising. Combining these advantages remains challenging: AR-to-dLLM conversion often fails to preserve seed-checkpoint capability, and hybrid-attention recurrent states and masking constraints make diffusion training and serving nontrivial. We present FLARE, a systematic conversion framework for hybrid-attention LLMs. Our analysis identifies transfer data quality as the primary determinant of capability preservation, outweighing loss formulation and attention-mask design. The resulting framework combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising. Starting from strong AR checkpoints with limited post-training data, FLARE is competitive with leading open-source dLLMs across model scales and delivers consistent throughput gains over open-source dLLM baselines in single-GPU concurrent serving. Our results further suggest that practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives, motivating joint design of data, objectives, architectures, and inference systems.

Problem

• diffusion language models
• hybrid attention
• autoregressive-to-diffusion conversion
• parallel generation
• capability preservation

Innovation

• diffusion language models
• hybrid attention
• parallel decoding
• capability preservation
• unified inference