Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers

2026-06-30

AI Summary

This work addresses the limitation of existing single-step diffusion distillation methods, which require teacher and student models to share the same latent space, thereby hindering knowledge transfer from high-capacity teachers to lightweight students such as Stable Diffusion 1.5. The study formalizes, for the first time, the cross-latent-space distillation problem and introduces a lightweight Bridge module that maps the student’s latent representations into the teacher’s space without modifying the student backbone. This module leverages the frozen student VAE decoder as a spatial prior combined with a learnable projector, optimized jointly via latent reconstruction and attention fidelity losses. The approach supports heterogeneous architectures and varying resolutions, achieving substantial performance gains—e.g., improving the HPSv3 score of SD 1.5 from 5.4 to 9.4—while preserving single-step inference, low latency, and ecosystem compatibility.

Abstract

Modern one-step diffusion models achieve impressive quality through distribution-based timestep distillation. Yet, they rely on a critical assumption: Teacher and Student must inhabit the same latent space. This Shared-Space constraint prevents knowledge transfer from modern high-capacity Teachers (e.g., SD 3.5 and Flux) into compact, deployment-friendly Students such as SD 1.5, whose latent resolution and VAE parameterization differ from the Teacher. We formalize this overlooked regime as Cross-Space Distillation, where Teacher and Student differ in both latent resolution and VAE space. To enable distillation under this mismatch, we introduce the Bridge, a lightweight latent interface that maps Student latents into the Teacher space without modifying the Student backbone. Bridge combines a frozen Student VAE decoder as a spatial prior with a compact learnable projector, and is trained with latent reconstruction and attention fidelity objectives for stable Teacher-space alignment. Across diverse modern Teachers, Bridge enables substantial gains for compact one-step Students; for example, it improves SD 1.5 from 5.4 to 9.4 HPSv3 while preserving one-step inference, low latency, and broad ecosystem compatibility. These results show that heterogeneous large Teachers can be distilled into efficient, deployable backbones through a lightweight latent-space interface.

Problem

• Cross-Space Distillation
• diffusion models
• knowledge distillation
• latent space mismatch
• one-step generation

Innovation

• Cross-Space Distillation
• Latent Bridge
• One-Step Diffusion
• Knowledge Distillation
• VAE Alignment