Optimizing Noise Schedules of Generative Models in High Dimensionss

2025-01-02

AI Summary

In high-dimensional spaces, suboptimal noise scheduling in diffusion models degrades generative fidelity due to misaligned noise annealing across feature scales. Method: We identify an inherent dichotomy and complementary deficiencies between VP and VE scheduling schemes in reconstructing high- and low-level data features; building upon a stochastic interpolation framework, we propose a task-adaptive dynamic noise scheduling method that enables joint modeling of multi-scale structural information for the first time. Contribution/Results: Theoretical analysis shows that our scheduler reduces the required number of probability flow ODE discretization steps from Θ(√d) to Θ(1). Empirical validation on Gaussian Mixture (GM) and Curie–Weiss (CW) theoretical models demonstrates full recovery of mode distributions and modal asymmetry, with sampling complexity independent of dimensionality—yielding substantial gains in high-dimensional generation efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent works have shown that diffusion models can undergo phase transitions, the resolution of which is needed for accurately generating samples. This has motivated the use of different noise schedules, the two most common choices being referred to as variance preserving (VP) and variance exploding (VE). Here we revisit these schedules within the framework of stochastic interpolants. Using the Gaussian Mixture (GM) and Curie-Weiss (CW) data distributions as test case models, we first investigate the effect of the variance of the initial noise distribution and show that VP recovers the low-level feature (the distribution of each mode) but misses the high-level feature (the asymmetry between modes), whereas VE performs oppositely. We also show that this dichotomy, which happens when denoising by a constant amount in each step, can be avoided by using noise schedules specific to VP and VE that allow for the recovery of both high- and low-level features. Finally we show that these schedules yield generative models for the GM and CW model whose probability flow ODE can be discretized using $Theta_d(1)$ steps in dimension $d$ instead of the $Theta_d(sqrt{d})$ steps required by constant denoising.

Problem

• Complex Spaces
• Generative Models
• Sample Accuracy

Innovation

• Variance Adjustment
• High-Dimensional Data Modeling
• Noise Control Optimization