LUIVITON: Learned Universal Interoperable VIrtual Try-ON

2025-09-05

AI Summary

This work addresses the challenge of fully automatic 3D virtual try-on for multi-layered garments on human-like characters with arbitrary poses and diverse anatomies—including cartoon, robotic, and alien forms. We propose a dual-correspondence learning framework that jointly maps both garment and body to an SMPL proxy model. Geometric learning enables precise part-to-whole garment-to-SMPL alignment, while multi-view consistent appearance features—guided by a pre-trained 2D foundation model—steer a diffusion model to achieve unsupervised, high-fidelity body-to-SMPL cross-domain correspondence. To our knowledge, this is the first end-to-end 3D virtual try-on system that is truly general-purpose, customizable, and fully automated—requiring no manual intervention, 2D garment cutouts, or mesh preprocessing. It natively supports non-manifold meshes, complex geometries, and post-hoc adjustments of garment size and material properties. The method exhibits strong generalization across unseen character types and high computational efficiency, making it suitable for real-world deployment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

We present LUIVITON, an end-to-end system for fully automated virtual try-on, capable of draping complex, multi-layer clothing onto diverse and arbitrarily posed humanoid characters. To address the challenge of aligning complex garments with arbitrary and highly diverse body shapes, we use SMPL as a proxy representation and separate the clothing-to-body draping problem into two correspondence tasks: 1) clothing-to-SMPL and 2) body-to-SMPL correspondence, where each has its unique challenges. While we address the clothing-to-SMPL fitting problem using a geometric learning-based approach for partial-to-complete shape correspondence prediction, we introduce a diffusion model-based approach for body-to-SMPL correspondence using multi-view consistent appearance features and a pre-trained 2D foundation model. Our method can handle complex geometries, non-manifold meshes, and generalizes effectively to a wide range of humanoid characters -- including humans, robots, cartoon subjects, creatures, and aliens, while maintaining computational efficiency for practical adoption. In addition to offering a fully automatic fitting solution, LUIVITON supports fast customization of clothing size, allowing users to adjust clothing sizes and material properties after they have been draped. We show that our system can produce high-quality 3D clothing fittings without any human labor, even when 2D clothing sewing patterns are not available.

Problem

• Automated draping of multi-layer clothing on diverse humanoid characters
• Aligning complex garments with arbitrary body shapes using SMPL proxy
• Handling non-manifold meshes and generalizing to various character types

Innovation

• SMPL proxy representation separates draping into dual correspondence tasks
• Geometric learning for clothing-to-SMPL partial-to-complete shape prediction
• Diffusion model with multi-view features for body-to-SMPL correspondence