Sig-DEG for Distillation: Making Diffusion Models Faster and Lighter

2025-08-23

AI Summary

Diffusion models achieve high-quality generation but suffer from slow inference, typically requiring thousands of discretization steps. To address this, we propose Signature-driven Differential Equation Generators (Sig-DEG), which efficiently approximate the reverse diffusion stochastic differential equation (SDE) on coarse temporal grids. Leveraging the cyclic structure of local signatures of Brownian motion and higher-order approximation theory, Sig-DEG integrates partial signature computation with a recursive sequence modeling network, casting knowledge distillation as a coarse-grained supervised learning task. Crucially, it achieves high-fidelity global solution approximation without fine-grained path simulation. Experiments on mainstream image generation benchmarks demonstrate that Sig-DEG reduces sampling steps by an order of magnitude—e.g., from 1000 to 100—while preserving FID and LPIPS scores comparable to those of the original diffusion models, thereby significantly improving inference efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Diffusion models have achieved state-of-the-art results in generative modelling but remain computationally intensive at inference time, often requiring thousands of discretization steps. To this end, we propose Sig-DEG (Signature-based Differential Equation Generator), a novel generator for distilling pre-trained diffusion models, which can universally approximate the backward diffusion process at a coarse temporal resolution. Inspired by high-order approximations of stochastic differential equations (SDEs), Sig-DEG leverages partial signatures to efficiently summarize Brownian motion over sub-intervals and adopts a recurrent structure to enable accurate global approximation of the SDE solution. Distillation is formulated as a supervised learning task, where Sig-DEG is trained to match the outputs of a fine-resolution diffusion model on a coarse time grid. During inference, Sig-DEG enables fast generation, as the partial signature terms can be simulated exactly without requiring fine-grained Brownian paths. Experiments demonstrate that Sig-DEG achieves competitive generation quality while reducing the number of inference steps by an order of magnitude. Our results highlight the effectiveness of signature-based approximations for efficient generative modeling.

Problem

• Reducing computational intensity of diffusion models inference
• Approximating backward diffusion process coarsely
• Enabling fast generation with fewer steps

Innovation

• Signature-based generator for diffusion distillation
• Recurrent structure for global SDE approximation
• Exact simulation without fine-grained Brownian paths