Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physics

2025-02-05

AI Summary

This work addresses the problem of reconstructing separable, physically plausible, and simulation-ready 3D clothing and human body models from a single in-the-wild image, enabling virtual try-on and dynamic cloth animation. We propose the first end-to-end framework that jointly leverages diffusion priors and differentiable physics-based simulation. Our method first employs a pre-trained image-to-pattern model to generate structural sewing patterns, which guide multi-view diffusion-based geometric reconstruction. Next, a differentiable cloth simulator performs joint geometry–physics optimization driven by the predicted sewing patterns. Finally, texture generation and human motion synthesis modules are integrated. The approach significantly improves image–geometry alignment accuracy, yielding high-fidelity, physically consistent, and directly simulatable customized dressed human models. It achieves a key breakthrough in dynamic clothing modeling from a single input image.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advances in large models have significantly advanced image-to-3D reconstruction. However, the generated models are often fused into a single piece, limiting their applicability in downstream tasks. This paper focuses on 3D garment generation, a key area for applications like virtual try-on with dynamic garment animations, which require garments to be separable and simulation-ready. We introduce Dress-1-to-3, a novel pipeline that reconstructs physics-plausible, simulation-ready separated garments with sewing patterns and humans from an in-the-wild image. Starting with the image, our approach combines a pre-trained image-to-sewing pattern generation model for creating coarse sewing patterns with a pre-trained multi-view diffusion model to produce multi-view images. The sewing pattern is further refined using a differentiable garment simulator based on the generated multi-view images. Versatile experiments demonstrate that our optimization approach substantially enhances the geometric alignment of the reconstructed 3D garments and humans with the input image. Furthermore, by integrating a texture generation module and a human motion generation module, we produce customized physics-plausible and realistic dynamic garment demonstrations. Project page: https://dress-1-to-3.github.io/

Problem

• Generate simulation-ready 3D garments from single images.
• Ensure garments are separable and physics-plausible.
• Enhance geometric alignment with input images.

Innovation

• Diffusion model for multi-view images
• Differentiable garment simulator refinement
• Integrated texture and motion generation