Guiding Token-Sparse Diffusion Models

2026-01-04

arXiv.org

AI Summary

This work addresses the degraded generation quality of sparsely trained diffusion models during inference, which stems from their insufficient response to classifier-free guidance (CFG). To overcome this limitation, the authors propose Sparse Guidance (SG), a novel approach that, for the first time, leverages token-level sparsity as a guidance signal in place of the conventional conditioning dropout mechanism. SG preserves the high variance inherent in conditional predictions while simultaneously enhancing both sample diversity and fidelity. Experimental results demonstrate that SG achieves a state-of-the-art FID of 1.58 on ImageNet-256 with 25% fewer FLOPs than baseline methods, and reduces computational cost by up to 58% at equivalent generation quality. Furthermore, when applied to a 2.5B-parameter text-to-image model, SG significantly improves compositional quality, human preference scores, and inference throughput.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.

Problem

• diffusion models
• token sparsity
• classifier-free guidance
• inference quality
• image synthesis

Innovation

• Sparse Guidance
• token-level sparsity
• diffusion models
• Classifier-free Guidance
• FLOP efficiency