Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

2026-03-15

AI Summary

This work addresses the limited diffusibility of latent spaces in latent diffusion models, which stems from a mismatch between the spectral distribution of latent variables and that of natural images, leading to degraded generation quality. The authors propose the Spectral Matching Hypothesis, introducing Encoder Spectral Matching (ESM) to enforce a flat power-law power spectral density in latent representations and Decoder Spectral Matching (DSM) to preserve frequency-wise semantic consistency. They establish, for the first time, a unified theoretical framework for latent diffusibility from a spectral perspective, explaining phenomena such as over-noising or over-smoothing in existing methods and unifying several approaches under this framework. Furthermore, they extend the spectral principle to representation alignment tasks (REPA). Experiments on CelebA and ImageNet demonstrate that the proposed method significantly outperforms current state-of-the-art techniques, validating the efficacy of spectral matching in enhancing diffusion model performance.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: (Large) Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

In this paper, we study the diffusability (learnability) of variational autoencoders (VAE) in latent diffusion. First, we show that pixel-space diffusion trained with an MSE objective is inherently biased toward learning low and mid spatial frequencies, and that the power-law power spectral density (PSD) of natural images makes this bias perceptually beneficial. Motivated by this result, we propose the \emph{Spectrum Matching Hypothesis}: latents with superior diffusability should (i) follow a flattened power-law PSD (\emph{Encoding Spectrum Matching}, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (\emph{Decoding Spectrum Matching}, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching provides a unified view that clarifies prior observations of over-noisy or over-smoothed latents, and interprets several recent methods as special cases (e.g., VA-VAE, EQ-VAE). Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available https://github.com/forever208/SpectrumMatching.

Problem

• diffusability
• latent diffusion
• spectrum matching
• power spectral density
• variational autoencoder

Innovation

• Spectrum Matching
• Power Spectral Density
• Latent Diffusion
• Frequency Alignment
• Representation Alignment