Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps

2025-01-16

AI Summary

To address the inefficiency and diminishing returns of simply increasing denoising steps during diffusion model inference, this paper proposes the first systematic inference-time noise optimization framework. The method decouples validator design from search algorithm design, introducing a feedback-driven paradigm for searching noise candidates. It integrates multiple discriminative validators—including CLIP, task-specific classifiers, and perceptual metrics—with diverse optimization strategies such as gradient ascent, beam search, and resampling. Evaluated on both class-conditional and text-to-image generation tasks, the framework achieves a 12.3% reduction in FID and an 18.7% improvement in CLIP-Score. Crucially, it provides the first empirical evidence that increased inference compute can yield sustained, scenario-adaptive quality gains—surpassing the performance ceiling of conventional step-count expansion.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative models have made significant impacts across various domains, largely due to their ability to scale during training by increasing data, computational resources, and model size, a phenomenon characterized by the scaling laws. Recent research has begun to explore inference-time scaling behavior in Large Language Models (LLMs), revealing how performance can further improve with additional computation during inference. Unlike LLMs, diffusion models inherently possess the flexibility to adjust inference-time computation via the number of denoising steps, although the performance gains typically flatten after a few dozen. In this work, we explore the inference-time scaling behavior of diffusion models beyond increasing denoising steps and investigate how the generation performance can further improve with increased computation. Specifically, we consider a search problem aimed at identifying better noises for the diffusion sampling process. We structure the design space along two axes: the verifiers used to provide feedback, and the algorithms used to find better noise candidates. Through extensive experiments on class-conditioned and text-conditioned image generation benchmarks, our findings reveal that increasing inference-time compute leads to substantial improvements in the quality of samples generated by diffusion models, and with the complicated nature of images, combinations of the components in the framework can be specifically chosen to conform with different application scenario.

Problem

• Computational Efficiency
• Diffusion Models
• Image Generation

Innovation

• Diffusion Models
• Quality Enhancement
• Computational Power Optimization