AIpparel: A Large Multimodal Generative Model for Digital Garments

2024-12-05

arXiv.org

AI Summary

Clothing design remains highly manual, with low automation and lengthy development cycles. To address this, we propose AIpparel—the first multimodal foundation model specifically designed for sewing pattern generation and interactive editing. Leveraging a large-scale dataset of over 120,000 garment instances annotated with text, images, and structured sewing patterns, we introduce a compact, sewing-pattern–specific tokenization scheme, enabling end-to-end modeling and real-time interactive editing of sewing patterns by large language models for the first time. By fine-tuning a state-of-the-art multimodal large model (LMM), we integrate a custom sewing pattern encoder, a cross-modal alignment mechanism, and a multi-task joint training strategy. AIpparel achieves state-of-the-art performance on text- and image-to-garment generation tasks and supports real-time interactive editing, significantly accelerating the digital apparel design workflow.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Humans and AI: Interaction Techniques and Devices

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Apparel is essential to human life, offering protection, mirroring cultural identities, and showcasing personal style. Yet, the creation of garments remains a time-consuming process, largely due to the manual work involved in designing them. To simplify this process, we introduce AIpparel, a large multimodal model for generating and editing sewing patterns. Our model fine-tunes state-of-the-art large multimodal models (LMMs) on a custom-curated large-scale dataset of over 120,000 unique garments, each with multimodal annotations including text, images, and sewing patterns. Additionally, we propose a novel tokenization scheme that concisely encodes these complex sewing patterns so that LLMs can learn to predict them efficiently. AIpparel achieves state-of-the-art performance in single-modal tasks, including text-to-garment and image-to-garment prediction, and enables novel multimodal garment generation applications such as interactive garment editing. The project website is at georgenakayama.github.io/AIpparel/.

Problem

• Simplifying garment design process using AI
• Generating sewing patterns from text or images
• Enabling interactive garment editing with multimodal AI

Innovation

• Fine-tunes LMMs on 120K multimodal garment dataset
• Novel tokenization for efficient sewing pattern encoding
• Enables text-to-garment and image-to-garment generation