Single-Branch Network Architectures to Close the Modality Gap in Multimodal Recommendation

2025-09-23

AI Summary

Multimodal recommendation suffers from degraded cold-start performance when user/item interaction histories are missing for certain modalities (i.e., modality dropout). Method: This paper proposes a single-branch network architecture that explicitly narrows cross-modal semantic gaps via weight sharing, stochastic modality sampling, and contrastive learning loss—thereby achieving tighter modality alignment in the embedding space. Unlike conventional multi-branch designs, this approach eliminates inconsistent optimization objectives across modalities and enhances generalization under modality dropout. Contribution/Results: Experiments on three public benchmarks demonstrate that our method achieves comparable performance to baselines in warm-start settings, while significantly outperforming multi-branch counterparts under modality dropout: it improves six accuracy metrics (e.g., Recall@20, NDCG@20) and four non-accuracy metrics (e.g., diversity, coverage) on average. These results validate its effectiveness and robustness for cold-start multimodal recommendation.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Traditional recommender systems rely on collaborative filtering, using past user-item interactions to help users discover new items in a vast collection. In cold start, i.e., when interaction histories of users or items are not available, content-based recommender systems use side information instead. Hybrid recommender systems (HRSs) often employ multimodal learning to combine collaborative and side information, which we jointly refer to as modalities. Though HRSs can provide recommendations when some modalities are missing, their quality degrades. In this work, we utilize single-branch neural networks equipped with weight sharing, modality sampling, and contrastive loss to provide accurate recommendations even in missing modality scenarios by narrowing the modality gap. We compare these networks with multi-branch alternatives and conduct extensive experiments on three datasets. Six accuracy-based and four beyond-accuracy-based metrics help assess the recommendation quality for the different training paradigms and their hyperparameters in warm-start and missing modality scenarios. We quantitatively and qualitatively study the effects of these different aspects on bridging the modality gap. Our results show that single-branch networks achieve competitive performance in warm-start scenarios and are significantly better in missing modality settings. Moreover, our approach leads to closer proximity of an item's modalities in the embedding space. Our full experimental setup is available at https://github.com/hcai-mms/single-branch-networks.

Problem

• Addressing performance degradation in hybrid recommender systems when modalities are missing
• Closing the modality gap between collaborative and content information in recommendations
• Improving recommendation accuracy in cold-start and missing modality scenarios

Innovation

• Single-branch neural networks with weight sharing
• Modality sampling and contrastive loss techniques
• Bridging modality gap for missing data