MSF: Efficient Diffusion Model Via Multi-Scale Latent Factorize

2025-01-23

AI Summary

To address the high computational cost and neglect of visual hierarchy in diffusion models for high-resolution image generation, this paper proposes a multi-scale latent factorization framework that decouples the generative process into two collaborative stages: semantic base modeling at low resolution and texture residual modeling at high resolution. We introduce a novel, vision-aware multi-scale latent decomposition paradigm—lighter than wavelet-based methods—that explicitly separates semantic structure from fine-grained texture. The framework integrates VAE latent space factorization, lightweight Transformers, and a staged diffusion strategy. Evaluated on ImageNet 256×256, our method achieves an FID of 2.2 and an Inception Score (IS) of 255.4, while reducing computational cost by 50% compared to baseline diffusion models. This demonstrates significant improvements in both generation quality and efficiency for high-resolution synthesis, striking a superior balance between fidelity and resource consumption.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion-based generative models have achieved remarkable progress in visual content generation. However, traditional diffusion models directly denoise the entire image from noisy inputs, disregarding the hierarchical structure present in visual signals. This method is computationally intensive, especially for high-resolution image generation. Signal processing often leverages hierarchical decompositions; for instance, Fourier analysis decomposes signals by frequency, while wavelet analysis captures localized frequency components, reflecting both spatial and frequency information simultaneously. Inspired by these principles, we propose a multiscale diffusion framework that generates hierarchical visual representations, which are subsequently integrated to form the final output. The diffusion model target, whether raw RGB pixels or latent features from a Variational Autoencoder, s divided into multiple components that each capture distinct spatial levels. The low-resolution component contains the primary informative signal, while higher-resolution components add high-frequency details, such as texture. This approach divides image generation into two stages: producing a low-resolution base signal, followed by a high-resolution residual signal. Both stages can be effectively modeled using simpler, lightweight transformer architectures compared to full-resolution generation. This decomposition is conceptually similar to wavelet decomposition but offers a more streamlined and intuitive design. Our method, termed MSF(short for Multi-Scale Factorization), achieves an FID of 2.2 and an IS of 255.4 on the ImageNet 256x256 benchmark, reducing computational costs by 50% compared to baseline methods.

Problem

• Hierarchical Structure
• Image Generation
• Diffusion Models

Innovation

• MSF Framework
• Hierarchical Image Generation
• Resource Efficiency