Unlocking the Potentials of Retrieval-Augmented Generation for Diffusion Language Models

2026-01-16

AI Summary

This work addresses the problem of response semantic drift (RSD) in diffusion language models (DLMs) when integrated with retrieval-augmented generation (RAG), where generated outputs often deviate from the original query, leading to reduced accuracy. The study is the first to identify and characterize this phenomenon and introduces SPREAD—a Semantic-Preserving Retrieval-Augmented Diffusion framework—that dynamically guides the denoising process using query relevance signals to anchor iterative generation to the original semantic intent. Experimental results demonstrate that SPREAD significantly improves both the factual accuracy and semantic consistency of generated responses, effectively mitigating semantic drift within the RAG-DLM paradigm.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Diffusion Language Models (DLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. However, the potential of Retrieval-Augmented Generation (RAG), which shows great successes for enhancing large language models (LLMs), has not been well explored, due to the fundamental difference between LLM and DLM decoding. To fill this critical gap, we systematically test the performance of DLMs within the RAG framework. Our findings reveal that DLMs coupled with RAG show promising potentials with stronger dependency on contextual information, but suffer from limited generation precision. We identify a key underlying issue: Response Semantic Drift (RSD), where the generated answer progressively deviates from the query's original semantics, leading to low precision content. We trace this problem to the denoising strategies in DLMs, which fail to maintain semantic alignment with the query throughout the iterative denoising process. To address this, we propose Semantic-Preserving REtrieval-Augmented Diffusion (SPREAD), a novel framework that introduces a query-relevance-guided denoising strategy. By actively guiding the denoising trajectory, SPREAD ensures the generation remains anchored to the query's semantics and effectively suppresses drift. Experimental results demonstrate that SPREAD significantly enhances the precision and effectively mitigates RSD of generated answers within the RAG framework.

Problem

• Diffusion Language Models
• Retrieval-Augmented Generation
• Semantic Drift
• Generation Precision
• Query Semantics

Innovation

• Retrieval-Augmented Generation
• Diffusion Language Models
• Semantic Drift
• Query-Relevance-Guided Denoising
• SPREAD