ModaFlow: Modality-Aware Flow Matching for High-Fidelity Virtual Try-On

2026-06-26

AI Summary

Existing virtual try-on methods struggle to simultaneously preserve garment semantic details and achieve accurate geometric alignment with the human body under large clothing deformations. To address this challenge, this work proposes a modality-aware flow matching framework that decouples visual and textual conditioning: visual embeddings provide structural guidance, while textual embeddings are integrated via classifier-free guidance with adaptive scaling and a zero-initialized velocity field. The approach introduces a modality-aware guidance mechanism, a regularization loss combining cosine similarity and perceptual flow discrimination, and a stochastic masking strategy, collectively enhancing alignment accuracy and robustness in unpaired settings. Experiments demonstrate state-of-the-art performance, with FID scores reduced by approximately 30% and 20% on paired and unpaired benchmarks, respectively.

Abstract

Image-based virtual try-on has emerged as a compelling task in e-commerce and augmented reality, yet existing methods struggle to simultaneously preserve fine garment semantics and adapt to diverse person body geometries under large clothing-body deformations. We present ModaFlow, a modality-aware flow-matching based framework for high-fidelity virtual try-on that achieves precise alignment between textual descriptions and garment appearance. Unlike prior methods that treat multimodal conditions uniformly, ModaFlow introduces a modality-aware guidance scheme: visual garment embeddings extracted by a pretrained image prompt adapter provide deterministic, persistent structural guidance, while textual embeddings generated from garment descriptions are controlled via classifier-free guidance (CFG) with adaptive scaling and zero-initialized velocity. To further enhance flow field accuracy, we propose two regularization losses, cosine similarity and perceptual flow discrimination, that jointly improve directional consistency and perceptual realism of the velocity field. Additionally, a mask manipulation strategy stochastically samples among box, transparent, and relaxed masks during training, simulating diverse occlusion scenarios and enabling robust inference under unpaired settings where only a box mask is available. Experiments show that ModaFlow achieves state-of-the-art results in both qualitative and quantitative evaluations, reducing FID by approximately 30% on paired and 20% on unpaired benchmarks.

Problem

• virtual try-on
• garment semantics
• body geometry
• clothing-body deformation
• high-fidelity

Innovation

• modality-aware
• flow matching
• classifier-free guidance
• regularization losses
• mask manipulation