Optimal Stopping in Latent Diffusion Models

2025-10-09

AI Summary

This work identifies the root cause of quality degradation in Latent Diffusion Models (LDMs) during late-stage sampling: intrinsic distortion from latent-space dimensionality compression—not numerical instability, as conventionally assumed. To address this, we establish a theoretical framework that explicitly links the optimal stopping time to key hyperparameters, including latent dimensionality and score-matching constraints—marking the first such characterization. Under Gaussian assumptions and a linear autoencoder model, we integrate score matching with diffusion dynamics to derive the universal principle: “low-dimensional latents require earlier stopping; high-dimensional ones permit later termination.” Experiments on both synthetic data and real images confirm that dynamically adapting the sampling steps per this principle significantly improves FID and LPIPS scores. Our study provides the first interpretable, computationally tractable, and practically applicable early-stopping theory and guideline for LDMs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We identify and analyze a surprising phenomenon of Latent Diffusion Models (LDMs) where the final steps of the diffusion can degrade sample quality. In contrast to conventional arguments that justify early stopping for numerical stability, this phenomenon is intrinsic to the dimensionality reduction in LDMs. We provide a principled explanation by analyzing the interaction between latent dimension and stopping time. Under a Gaussian framework with linear autoencoders, we characterize the conditions under which early stopping is needed to minimize the distance between generated and target distributions. More precisely, we show that lower-dimensional representations benefit from earlier termination, whereas higher-dimensional latent spaces require later stopping time. We further establish that the latent dimension interplays with other hyperparameters of the problem such as constraints in the parameters of score matching. Experiments on synthetic and real datasets illustrate these properties, underlining that early stopping can improve generative quality. Together, our results offer a theoretical foundation for understanding how the latent dimension influences the sample quality, and highlight stopping time as a key hyperparameter in LDMs.

Problem

• Early stopping improves latent diffusion model sample quality
• Latent dimension interacts with optimal diffusion stopping time
• Lower-dimensional representations require earlier termination than higher-dimensional ones

Innovation

• Early stopping improves latent diffusion model quality
• Latent dimension determines optimal diffusion stopping time
• Lower-dimensional representations benefit from earlier termination