Denoising, Fast and Slow: Difficulty-Aware Adaptive Sampling for Image Generation

2026-04-21

AI Summary

Existing diffusion and flow models allocate computational resources uniformly during image generation, disregarding the varying denoising difficulty across image regions. This work proposes Patch Forcing, a novel framework that enables controllable, patch-wise timestep sampling during training for the first time. It introduces a lightweight difficulty prediction head to estimate local denoising complexity and integrates spatiotemporal noise scheduling with an adaptive sampling strategy to dynamically allocate computation: simpler regions are denoised earlier to assist in generating more challenging ones, while preventing information leakage. The method achieves state-of-the-art performance on class-conditional ImageNet generation, seamlessly incorporates representation alignment and guidance techniques, and successfully extends to text-to-image synthesis tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Diffusion- and flow-based models usually allocate compute uniformly across space, updating all patches with the same timestep and number of function evaluations. While convenient, this ignores the heterogeneity of natural images: some regions are easy to denoise, whereas others benefit from more refinement or additional context. Motivated by this, we explore patch-level noise scales for image synthesis. We find that naively varying timesteps across image tokens performs poorly, as it exposes the model to overly informative training states that do not occur at inference. We therefore introduce a timestep sampler that explicitly controls the maximum patch-level information available during training, and show that moving from global to patch-level timesteps already improves image generation over standard baselines. By further augmenting the model with a lightweight per-patch difficulty head, we enable adaptive samplers that allocate compute dynamically where it is most needed. Combined with noise levels varying over both space and diffusion time, this yields Patch Forcing (PF), a framework that advances easier regions earlier so they can provide context for harder ones. PF achieves superior results on class-conditional ImageNet, remains orthogonal to representation alignment and guidance methods, and scales to text-to-image synthesis. Our results suggest that patch-level denoising schedules provide a promising foundation for adaptive image generation.

Problem

• denoising
• adaptive sampling
• image generation
• diffusion models
• patch-level difficulty

Innovation

• adaptive sampling
• patch-level denoising
• diffusion models
• noise scheduling
• difficulty-aware generation