InstructVVT: Instruction-Driven Video Virtual Try-On without Auxiliary Spatial Priors

2026-08-14

AI Summary

This study addresses the limitations of video virtual try-on methods that rely on error-prone spatial priors and struggle to capture human preferences. We propose an instruction-driven framework that eliminates the need for inference-time spatial priors by integrating a Diffusion Transformer with multimodal large language model semantic reasoning. Specifically, we introduce a dual-level reference conditioning mechanism and a try-on-specific reward model, DiffusionNFT, to achieve robust preference alignment. Experimental results demonstrate that this approach significantly outperforms existing open-source baselines in fidelity, structural preservation, and temporal consistency. Furthermore, the proposed method offers streamlined inference control, effectively overcoming the inherent drawbacks of conventional approaches dependent on fragile spatial guidance.

Abstract

Video virtual try-on is a highly constrained editing task requiring the precise replacement of a target person's clothing while strictly preserving the original video's spatial structure and temporal dynamics. Existing methods heavily rely on auxiliary handcrafted spatial priors (e.g., masks, poses) for editing control. However, these priors are prone to failure in unconstrained real-world videos and often compress rich visual context into incomplete structural signals. Furthermore, standard reconstruction objectives fail to fully capture try-on-specific human preferences. To address these challenges, we propose InstructVVT, an instruction-driven and reference-guided video virtual try-on framework based on a Diffusion Transformer (DiT) that operates without inference-time spatial priors. Our core insight is to recover fine-grained control directly from the input triplet (source video, reference garment, and instruction) via a dual-level reference conditioning scheme. Specifically, an MLLM infers semantic edit tokens for target disambiguation and structural preservation, while a lightweight conditioning pathway explicitly injects fine-grained visual garment details. Finally, we design a try-on-specific reward and utilize the DiffusionNFT algorithm to align the model with human preferences. Extensive experiments on ViViD-S and TripVVT-Bench demonstrate that InstructVVT outperforms state-of-the-art open-source methods in garment fidelity, structural preservation, and temporal consistency, despite requiring fewer inference-time controls.

Problem

• Video Virtual Try-On
• Spatial Priors
• Human Preferences
• Unconstrained Videos

Innovation

• Instruction-Driven VVT
• No Spatial Priors
• Dual-Level Reference Conditioning
• DiffusionNFT
• Diffusion Transformer