SNOOPI: Supercharged One-step Diffusion Distillation with Proper Guidance

2024-12-03

arXiv.org

AI Summary

To address weak controllability, ineffective negative prompting, and unstable distillation in single-step diffusion models for real-time image synthesis, this paper introduces two core innovations: (1) Probabilistic Guidance with Scaled Binary classifier-free guidance (PG-SB), which enhances distillation stability; and (2) a training-free Negative-Attention Steering via Attention (NASA) mechanism, the first to enable effective negative prompt response in single-step models. Our method integrates Variational Score Distillation (VSD) with a Cross-Attention-driven “Negative-Away Steer” mechanism, preserving one-step generation efficiency while significantly improving attribute controllability. On the HPSv2 benchmark, our approach achieves 31.08, establishing a new state-of-the-art for single-step diffusion models. It also surpasses SwiftBrushv2 and the teacher model across all major metrics—including FID and CLIP Score—demonstrating superior fidelity, alignment, and prompt adherence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent approaches have yielded promising results in distilling multi-step text-to-image diffusion models into one-step ones. The state-of-the-art efficient distillation technique, i.e., SwiftBrushv2 (SBv2), even surpasses the teacher model's performance with limited resources. However, our study reveals its instability when handling different diffusion model backbones due to using a fixed guidance scale within the Variational Score Distillation (VSD) loss. Another weakness of the existing one-step diffusion models is the missing support for negative prompt guidance, which is crucial in practical image generation. This paper presents SNOOPI, a novel framework designed to address these limitations by enhancing the guidance in one-step diffusion models during both training and inference. First, we effectively enhance training stability through Proper Guidance-SwiftBrush (PG-SB), which employs a random-scale classifier-free guidance approach. By varying the guidance scale of both teacher models, we broaden their output distributions, resulting in a more robust VSD loss that enables SB to perform effectively across diverse backbones while maintaining competitive performance. Second, we propose a training-free method called Negative-Away Steer Attention (NASA), which integrates negative prompts into one-step diffusion models via cross-attention to suppress undesired elements in generated images. Our experimental results show that our proposed methods significantly improve baseline models across various metrics. Remarkably, we achieve an HPSv2 score of 31.08, setting a new state-of-the-art benchmark for one-step diffusion models.

Problem

• One-step diffusion models sacrifice controllability for generation speed
• Negative prompting causes blending artifacts in one-step image synthesis
• Existing methods lack efficient attribute control in real-time generators

Innovation

• Negative-Away Steer Attention for one-step diffusion
• Leverages cross-attention to suppress undesired attributes
• Seamlessly integrates into timestep distillation frameworks