On the Limits of Latent Reuse in Diffusion Models

2026-05-13

AI Summary

This study investigates the reliability of reusing low-dimensional latent spaces from a source domain in diffusion models under distribution shift. By analyzing the geometric relationship between source and target domain subspaces, the work quantifies error sources in latent reuse through two key factors: principal angles between subspaces and noise amplification effects, and derives an upper bound on the approximation error of the target-domain score function. Building on this analysis, the authors propose a strategy for selecting shared latent dimensions during hybrid training and establish a theoretical criterion for latent space reusability, precisely delineating when direct transfer is feasible versus when learning a shared representation is necessary. This work provides both a geometric perspective and theoretical foundation for efficient cross-domain adaptation of diffusion models.

Abstract

Diffusion models are often trained in low-dimensional latent spaces, which are then reused for related but shifted datasets. In this work, we study when such latent reuse remains reliable under distribution shift. We consider a source-target setting in which both datasets are approximately low-dimensional but may lie near different subspaces. We show that freezing and reusing a source latent space induces a target-domain score error governed by two quantities: the principal-angle misalignment between the source and target subspaces, and the target ambient noise amplified by the diffusion time scale. Motivated by these limits, we further study mixed source-target training and characterize how the required shared latent dimension depends on the relative geometry of the two distributions. Our results provide theoretical guidance on when latent reuse is reliable and when learning a shared representation may be necessary.

Problem

• latent reuse
• diffusion models
• distribution shift
• subspace misalignment
• score error

Innovation

• latent reuse
• diffusion models
• distribution shift
• principal angles
• shared representation