FashionM3: Multimodal, Multitask, and Multiround Fashion Assistant based on Unified Vision-Language Model

2025-04-24

AI Summary

To address low efficiency and weak interactivity in personalized outfit recommendation for fashion retail, this work proposes the first unified multimodal vision-language model (VLM) architecture tailored for multi-turn dialogue, supporting four core tasks: outfit recommendation, item substitution, garment image generation, and virtual try-on. Methodologically, we introduce FashionRec—a large-scale, context-rich dataset comprising 330K samples spanning basic, personalized, and substitution-based recommendations; design a context-aware, progressive cross-task joint modeling framework; and integrate instruction tuning, diffusion-based image generation, and physics-simulation-driven virtual try-on. Experiments demonstrate significant improvements over state-of-the-art methods across multiple fashion understanding and generation benchmarks. User studies show a 32% increase in recommendation satisfaction and an average interaction naturalness score of 4.6/5, confirming strong practical applicability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Fashion styling and personalized recommendations are pivotal in modern retail, contributing substantial economic value in the fashion industry. With the advent of vision-language models (VLM), new opportunities have emerged to enhance retailing through natural language and visual interactions. This work proposes FashionM3, a multimodal, multitask, and multiround fashion assistant, built upon a VLM fine-tuned for fashion-specific tasks. It helps users discover satisfying outfits by offering multiple capabilities including personalized recommendation, alternative suggestion, product image generation, and virtual try-on simulation. Fine-tuned on the novel FashionRec dataset, comprising 331,124 multimodal dialogue samples across basic, personalized, and alternative recommendation tasks, FashionM3 delivers contextually personalized suggestions with iterative refinement through multiround interactions. Quantitative and qualitative evaluations, alongside user studies, demonstrate FashionM3's superior performance in recommendation effectiveness and practical value as a fashion assistant.

Problem

• Enhancing fashion retail with multimodal vision-language interactions
• Providing personalized and iterative fashion outfit recommendations
• Enabling multitask capabilities like virtual try-on and image generation

Innovation

• Unified vision-language model for fashion tasks
• Multimodal multitask multiround fashion assistant
• Fine-tuned on 331K FashionRec dialogue samples