CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

2026-07-10

AI Summary

Existing virtual try-on systems struggle to support fine-grained user control over garment wearing styles, such as tightness, aesthetic preferences, and body-relative positioning. This work reframes virtual try-on as an image editing task and introduces CtrlVTON, the first controllable virtual try-on framework that enables explicit manipulation of garment layout through pixel-level segmentation masks. To this end, we formulate a novel task termed Visual-Instance-Prompt Segmentation and propose the VIP-SAM model to generate high-fidelity masks. Experimental results demonstrate that CtrlVTON significantly outperforms state-of-the-art commercial systems in adherence to desired garment layouts while maintaining comparable garment fidelity, achieving state-of-the-art performance on both key evaluation metrics.

Abstract

Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.

Problem

• Virtual Try-On
• User Control
• Garment Layout
• Instance Segmentation
• Image Editing

Innovation

• Visual-Instance-Prompt Segmentation
• VIP-SAM
• CtrlVTON
• controllable virtual try-on
• instance-level segmentation