A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation

2025-11-10

AI Summary

This work addresses two core tasks in fashion recommendation: outfit compatibility prediction and complementary item retrieval. Methodologically, we propose a unified multimodal framework built upon CLIP encoders to jointly extract visual and textual features. We introduce two learnable tokens—“compatibility token” to model holistic outfit compatibility and “target-item token” to capture fine-grained complementarity requirements—and employ a Transformer to model cross-modal relationships. The framework is optimized via contrastive learning and end-to-end training for robust multimodal representation learning. Our key contribution is the first unified multimodal architecture that explicitly decouples global compatibility semantics from local complementarity semantics across both tasks. On the Polyvore dataset, our approach achieves an AUC of 0.95 for compatibility prediction and 69.24% accuracy on the Fill-in-the-Blank task, significantly outperforming state-of-the-art methods.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems

Abstract

The rapid expansion of online fashion platforms has created an increasing demand for intelligent recommender systems capable of understanding both visual and textual cues. This paper proposes a hybrid multimodal deep learning framework for fashion recommendation that jointly addresses two key tasks: outfit compatibility prediction and complementary item retrieval. The model leverages the visual and textual encoders of the CLIP architecture to obtain joint latent representations of fashion items, which are then integrated into a unified feature vector and processed by a transformer encoder. For compatibility prediction, an"outfit token"is introduced to model the holistic relationships among items, achieving an AUC of 0.95 on the Polyvore dataset. For complementary item retrieval, a"target item token"representing the desired item description is used to retrieve compatible items, reaching an accuracy of 69.24% under the Fill-in-the-Blank (FITB) metric. The proposed approach demonstrates strong performance across both tasks, highlighting the effectiveness of multimodal learning for fashion recommendation.

Problem

• Developing a hybrid multimodal framework for intelligent fashion recommendation systems
• Addressing outfit compatibility prediction using visual and textual item representations
• Solving complementary item retrieval through multimodal deep learning techniques

Innovation

• Uses CLIP encoders for multimodal fashion representations
• Introduces outfit token for compatibility prediction modeling
• Employs target item token for complementary item retrieval