Shape-Guided Clothing Warping for Virtual Try-On

2024-10-28

ACM Multimedia

AI Summary

Existing virtual try-on methods often suffer from insufficient fine-grained control over garment deformation, leading to body shape inconsistency and limb-region distortion. To address this, we propose SCW-VTON, a shape-guided virtual try-on framework featuring a novel dual-path garment deformation module: a shape path that incorporates global human geometric constraints, and a flow path that integrates Thin-Plate Spline (TPS) enhancement with appearance-based optical flow estimation. Additionally, we design a mask-modelling-driven limb reconstruction network that fuses limb-texture priors for precise restoration of distorted regions. Extensive experiments demonstrate that SCW-VTON consistently outperforms state-of-the-art methods—including VITON and ACGPN—on standard benchmarks. It significantly improves garment-body shape consistency and limb-detail naturalness, achieving superior performance in both qualitative and quantitative evaluations.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Manipulation

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Image-based virtual try-on aims to seamlessly fit in-shop clothing to a person image while maintaining pose consistency. Existing methods commonly employ the thin plate spline (TPS) transformation or appearance flow to deform in-shop clothing for aligning with the person's body. Despite their promising performance, these methods often lack precise control over fine details, leading to inconsistencies in shape between clothing and the person's body as well as distortions in exposed limb regions. To tackle these challenges, we propose a novel shape-guided clothing warping method for virtual try-on, dubbed SCW-VTON, which incorporates global shape constraints and additional limb textures to enhance the realism and consistency of the warped clothing and try-on results. To integrate global shape constraints for clothing warping, we devise a dual-path clothing warping module comprising a shape path and a flow path. The former path captures the clothing shape aligned with the person's body, while the latter path leverages the mapping between the pre- and post-deformation of the clothing shape to guide the estimation of appearance flow. Furthermore, to alleviate distortions in limb regions of try-on results, we integrate detailed limb guidance by developing a limb reconstruction network based on masked image modeling. Through the utilization of SCW-VTON, we are able to generate try-on results with enhanced clothing shape consistency and precise control over details. Extensive experiments demonstrate the superiority of our approach over state-of-the-art methods both qualitatively and quantitatively. The code is available at https://github.com/xyhanHIT/SCW-VTON.

Problem

• Improving clothing shape consistency in virtual try-on
• Reducing distortions in exposed limb regions
• Enhancing realism with global shape constraints

Innovation

• Shape-guided dual-path clothing warping module
• Limb reconstruction network for detail control
• Global shape constraints enhance clothing consistency