Rethinking Garment Conditioning in Diffusion-based Virtual Try-On

2025-11-24

AI Summary

To address the high computational cost of dual-UNet architectures and the low fidelity of single-UNet designs in diffusion-based virtual try-on (VTON), this paper proposes Re-CatVTON—a lightweight single-UNet framework. Our method introduces three key innovations: (1) a spatially concatenated garment conditioning injection mechanism, eliminating redundant feature fusion; (2) direct injection of latent representations derived from real garments to mitigate error accumulation during denoising; and (3) a context-aware modified classifier-free guidance strategy for enhanced fine-grained control. Experiments demonstrate that Re-CatVTON significantly outperforms CatVTON on FID, KID, and LPIPS metrics, achieving quality comparable to the dual-UNet model Leffa. Moreover, it reduces GPU memory consumption by 37% and accelerates inference by 2.1×, striking an optimal trade-off between efficiency and visual fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Virtual Try-On (VTON) is the task of synthesizing an image of a person wearing a target garment, conditioned on a person image and a garment image. While diffusion-based VTON models featuring a Dual UNet architecture demonstrate superior fidelity compared to single UNet models, they incur substantial computational and memory overhead due to their heavy structure. In this study, through visualization analysis and theoretical analysis, we derived three hypotheses regarding the learning of context features to condition the denoising process. Based on these hypotheses, we developed Re-CatVTON, an efficient single UNet model that achieves high performance. We further enhance the model by introducing a modified classifier-free guidance strategy tailored for VTON's spatial concatenation conditioning, and by directly injecting the ground-truth garment latent derived from the clean garment latent to prevent the accumulation of prediction error. The proposed Re-CatVTON significantly improves performance compared to its predecessor (CatVTON) and requires less computation and memory than the high-performance Dual UNet model, Leffa. Our results demonstrate improved FID, KID, and LPIPS scores, with only a marginal decrease in SSIM, establishing a new efficiency-performance trade-off for single UNet VTON models.

Problem

• Reducing computational and memory overhead in diffusion-based virtual try-on models
• Improving garment conditioning efficiency while maintaining high synthesis fidelity
• Optimizing single UNet architecture to match dual UNet model performance

Innovation

• Single UNet architecture for efficient virtual try-on
• Modified classifier-free guidance for spatial conditioning
• Direct ground-truth garment latent injection