Non-Markovian Discrete Diffusion with Causal Language Models

2025-02-13

AI Summary

Discrete diffusion models suffer from limited expressiveness and inferior performance compared to autoregressive causal language models (CLMs) in modeling structured sequences. To address this, we propose CaDDi—the first non-Markovian discrete diffusion framework. Methodologically, CaDDi integrates causal masking into the diffusion process, enabling denoising trajectories to inherently capture sequential dependencies; it formally unifies CLMs as a special case of the framework, allowing pretrained large language models (LLMs) to be directly incorporated without architectural modification. The approach combines non-Markovian transition kernels, discrete sequence denoising, and causal-constrained modeling. Experiments demonstrate that CaDDi significantly outperforms existing discrete diffusion models on both natural language and biological sequence tasks, while substantially narrowing the performance gap with state-of-the-art autoregressive LLMs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Discrete diffusion models have emerged as a flexible and controllable paradigm for structured sequence modeling, yet they still lag behind causal language models in expressiveness. To bridge the gap between two paradigms, we introduce CaDDi, a causal discrete diffusion model that unifies sequential and temporal modeling within a non-Markovian diffusion framework. Unlike conventional diffusion models that operate step by step with no access to prior states, CaDDi integrates the temporal trajectory, enabling more expressive and controllable generation. Our approach also treats causal language models as a special case, allowing seamless adoption of pretrained large language models (LLMs) for discrete diffusion without the need for architectural modifications. Empirically, we demonstrate that CaDDi outperforms state-of-the-art discrete diffusion models on both natural language and biological sequence tasks, narrowing the gap between diffusion-based methods and large-scale autoregressive transformers.

Problem

• Bridges gap between diffusion and causal models
• Enhances sequence modeling expressiveness and control
• Integrates large language models seamlessly

Innovation

• Unifies sequential and temporal modeling
• Integrates temporal trajectory for generation
• Adopts pretrained LLMs without modifications