Diffusion Models with Adaptive Negative Sampling Without External Resources

2025-08-04

AI Summary

Diffusion models (DMs) suffer from insufficient prompt adherence in text-to-image generation; existing negative prompting approaches rely on handcrafted prompts, incur information loss, and exhibit poor generalization. This paper establishes, for the first time, an intrinsic connection between negative prompting and classifier-free guidance (CFG), and proposes a training-free, resource-free adaptive negative sampling method: it implicitly models negation semantics via the CFG mechanism and dynamically aligns with negative concepts during sampling—thereby enhancing fidelity to the positive prompt. Crucially, the method eliminates explicit negative prompt inputs entirely, offering universal applicability and plug-and-play deployment. Extensive evaluations across multiple benchmarks demonstrate substantial improvements over strong baselines; in human preference studies, our method is selected twice as often as competing approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Diffusion models (DMs) have demonstrated an unparalleled ability to create diverse and high-fidelity images from text prompts. However, they are also well-known to vary substantially regarding both prompt adherence and quality. Negative prompting was introduced to improve prompt compliance by specifying what an image must not contain. Previous works have shown the existence of an ideal negative prompt that can maximize the odds of the positive prompt. In this work, we explore relations between negative prompting and classifier-free guidance (CFG) to develop a sampling procedure, {it Adaptive Negative Sampling Without External Resources} (ANSWER), that accounts for both positive and negative conditions from a single prompt. This leverages the internal understanding of negation by the diffusion model to increase the odds of generating images faithful to the prompt. ANSWER is a training-free technique, applicable to any model that supports CFG, and allows for negative grounding of image concepts without an explicit negative prompts, which are lossy and incomplete. Experiments show that adding ANSWER to existing DMs outperforms the baselines on multiple benchmarks and is preferred by humans 2x more over the other methods.

Problem

• Enhancing prompt adherence in diffusion models without external resources
• Improving image quality by adaptive negative sampling techniques
• Eliminating need for explicit negative prompts in diffusion models

Innovation

• Adaptive negative sampling without external resources
• Training-free technique for diffusion models
• Leverages internal understanding of negation