FastFit: Accelerating Multi-Reference Virtual Try-On via Cacheable Diffusion Models

2025-08-28

AI Summary

Virtual try-on faces two key challenges: insufficient support for multi-reference garments (including accessories) and low inference efficiency. To address these, we propose CacheDiffusion—the first cacheable diffusion framework for virtual try-on—featuring a semi-attention mechanism and class embeddings that replace timestep embeddings, thereby decoupling reference feature encoding from the denoising process and enabling lossless cross-step feature reuse. Its cache-aware diffusion architecture significantly reduces redundant computation. We further introduce DressCode-MR, the first large-scale multi-reference virtual try-on dataset, with annotation quality enhanced via expert models and human feedback. Extensive experiments on VITON-HD, DressCode, and DressCode-MR demonstrate state-of-the-art performance: CacheDiffusion achieves superior fidelity (measured by FID, LPIPS) and accelerates average inference speed by 3.5× over prior methods. To our knowledge, it is the first approach to simultaneously achieve high fidelity, high efficiency, and robust multi-reference compatibility in virtual try-on.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Despite its great potential, virtual try-on technology is hindered from real-world application by two major challenges: the inability of current methods to support multi-reference outfit compositions (including garments and accessories), and their significant inefficiency caused by the redundant re-computation of reference features in each denoising step. To address these challenges, we propose FastFit, a high-speed multi-reference virtual try-on framework based on a novel cacheable diffusion architecture. By employing a Semi-Attention mechanism and substituting traditional timestep embeddings with class embeddings for reference items, our model fully decouples reference feature encoding from the denoising process with negligible parameter overhead. This allows reference features to be computed only once and losslessly reused across all steps, fundamentally breaking the efficiency bottleneck and achieving an average 3.5x speedup over comparable methods. Furthermore, to facilitate research on complex, multi-reference virtual try-on, we introduce DressCode-MR, a new large-scale dataset. It comprises 28,179 sets of high-quality, paired images covering five key categories (tops, bottoms, dresses, shoes, and bags), constructed through a pipeline of expert models and human feedback refinement. Extensive experiments on the VITON-HD, DressCode, and our DressCode-MR datasets show that FastFit surpasses state-of-the-art methods on key fidelity metrics while offering its significant advantage in inference efficiency.

Problem

• Enabling multi-reference outfit compositions for virtual try-on
• Reducing redundant feature recomputation in diffusion models
• Accelerating inference speed while maintaining high fidelity

Innovation

• Cacheable diffusion architecture for feature reuse
• Semi-Attention mechanism with class embeddings
• Decouples reference encoding from denoising process