Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models

2026-04-12

AI Summary

This study addresses the poorly understood failure mechanisms underlying hallucinations in diffusion-based large language models (dLLMs). Through controlled comparative experiments that hold architecture, scale, and pretrained weights constant, the work systematically examines hallucination behaviors in dLLMs relative to autoregressive models and investigates how inference-time computation affects generation quality. The analysis reveals, for the first time, that dLLMs are significantly more prone to hallucinations than their autoregressive counterparts, identifying distinct failure modes—including premature termination, incomplete denoising, and context intrusion—and demonstrates that non-autoregressive decoding retains potential for iterative refinement. Although dLLMs achieve comparable performance on general tasks, their reliability remains substantially inferior.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While Diffusion Large Language Models (dLLMs) have emerged as a promising non-autoregressive paradigm comparable to autoregressive (AR) models, their faithfulness, specifically regarding hallucination, remains largely underexplored. To bridge this gap, we present the first controlled comparative study to evaluate hallucination patterns in dLLMs. Our results demonstrate that current dLLMs exhibit a higher propensity for hallucination than AR counterparts controlled for architecture, scale, and pre-training weights. Furthermore, an analysis of inference-time compute reveals divergent dynamics: while quasi-autoregressive generation suffers from early saturation, non-sequential decoding unlocks potential for continuous refinement. Finally, we identify distinct failure modes unique to the diffusion process, including premature termination, incomplete denoising, and context intrusion. Our findings underscore that although dLLMs have narrowed the performance gap on general tasks, their distinct hallucination mechanisms pose a critical challenge to model reliability. Our code is available at https://github.com/ZeroLoss-Lab/Lost-in-Diffusion

Problem

• hallucination
• diffusion language models
• failure modes
• faithfulness
• non-autoregressive generation

Innovation

• diffusion language models
• hallucination
• non-autoregressive generation
• failure modes
• controlled comparison