Simple Guidance Mechanisms for Discrete Diffusion Models

2024-12-13

arXiv.org

AI Summary

This work addresses the challenge of controllable diffusion-based generation over discrete data—where existing classifier-guidance techniques designed for continuous domains cannot be directly applied. We propose the first rigorously derived classifier-free and classifier-guidance frameworks specifically for discrete domains. Our method introduces a discrete diffusion model based on a uniform noise schedule and establishes a continuous-time variational lower bound, enabling efficient differentiable editing and strong controllability. The framework unifies discrete modeling, noise scheduling, variational inference, and guidance mechanism design. Evaluated on gene sequence generation, small-molecule design, and discrete image synthesis, our approach significantly outperforms autoregressive and state-of-the-art diffusion-based baselines, achieving new SOTA performance in both sample quality and inference speed.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Diffusion models for continuous data gained widespread adoption owing to their high quality generation and control mechanisms. However, controllable diffusion on discrete data faces challenges given that continuous guidance methods do not directly apply to discrete diffusion. Here, we provide a straightforward derivation of classifier-free and classifier-based guidance for discrete diffusion, as well as a new class of diffusion models that leverage uniform noise and that are more guidable because they can continuously edit their outputs. We improve the quality of these models with a novel continuous-time variational lower bound that yields state-of-the-art performance, especially in settings involving guidance or fast generation. Empirically, we demonstrate that our guidance mechanisms combined with uniform noise diffusion improve controllable generation relative to autoregressive and diffusion baselines on several discrete data domains, including genomic sequences, small molecule design, and discretized image generation.

Problem

• Develop guidance for discrete diffusion
• Enhance controllable discrete data generation
• Improve performance with uniform noise models

Innovation

• Discrete diffusion guidance mechanisms
• Uniform noise diffusion models
• Continuous-time variational lower bound