Accelerated Diffusion Models via Speculative Sampling

2025-01-09

AI Summary

To address the low sampling efficiency of diffusion models, this work pioneers the extension of speculative sampling to continuous vector spaces, proposing a training-free, plug-and-play lightweight draft strategy. Unlike prior approaches, it requires no auxiliary draft model training; instead, it leverages a Markov chain mechanism coupled with function evaluation optimization to guarantee end-to-end exact sampling strictly conforming to the target diffusion model’s distribution. Experiments across multiple mainstream diffusion models demonstrate an average 50% reduction in function evaluations, yielding substantial speedup while preserving perfect fidelity to the target distribution. The core contributions are: (i) the first theoretically grounded and practically validated adaptation of speculative sampling to continuous-space diffusion models; and (ii) a novel draft-generation paradigm ensuring distributional consistency with zero training overhead.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Speculative sampling is a popular technique for accelerating inference in Large Language Models by generating candidate tokens using a fast draft model and accepting or rejecting them based on the target model's distribution. While speculative sampling was previously limited to discrete sequences, we extend it to diffusion models, which generate samples via continuous, vector-valued Markov chains. In this context, the target model is a high-quality but computationally expensive diffusion model. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model and is applicable out of the box to any diffusion model. Our experiments demonstrate significant generation speedup on various diffusion models, halving the number of function evaluations, while generating exact samples from the target model.

Problem

• Large Language Models
• Generation Speed
• Diffusion Models

Innovation

• Speculative Sampling
• Diffusion Models
• Speed Optimization