Are Multimodal Embeddings Truly Beneficial for Recommendation? A Deep Dive into Whole vs. Individual Modalities

2025-08-10

AI Summary

Does multimodal embedding genuinely improve recommendation performance? This work systematically investigates the actual contributions of textual and visual modalities in modern multimodal recommender systems. Method: We propose a “modality ablation” strategy—systematically evaluating unimodal (text-only or image-only) versus full-modal performance across 14 state-of-the-art models, using pretrained feature extractors and either graph-based or simple fusion baselines; robustness is further validated via constant/noise injection. Contribution/Results: Text-only representations achieve performance on par with—or even surpass—that of full multimodal models, whereas image-only inputs yield negligible gains. While sophisticated graph-based fusion models show notable improvements, most lightweight baselines exhibit marginal benefits from multimodality. Our findings delineate the practical effectiveness boundary of multimodal fusion, challenging the implicit assumption that “more modalities always yield better performance.” The study provides empirical grounding for model design and advocates principled simplicity in multimodal recommendation.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal recommendation (MMRec) has emerged as a mainstream paradigm, typically leveraging text and visual embeddings extracted from pre-trained models such as Sentence-BERT, Vision Transformers, and ResNet. This approach is founded on the intuitive assumption that incorporating multimodal embeddings can enhance recommendation performance. However, despite its popularity, this assumption lacks comprehensive empirical verification. This presents a critical research gap. To address it, we pose the central research question of this paper: Are multimodal embeddings truly beneficial for recommendation? To answer this question, we conduct a large-scale empirical study examining the role of text and visual embeddings in modern MMRec models, both as a whole and individually. Specifically, we pose two key research questions: (1) Do multimodal embeddings as a whole improve recommendation performance? (2) Is each individual modality - text and image - useful when used alone? To isolate the effect of individual modalities - text or visual - we employ a modality knockout strategy by setting the corresponding embeddings to either constant values or random noise. To ensure the scale and comprehensiveness of our study, we evaluate 14 widely used state-of-the-art MMRec models. Our findings reveal that: (1) multimodal embeddings generally enhance recommendation performance - particularly when integrated through more sophisticated graph-based fusion models. Surprisingly, commonly adopted baseline models with simple fusion schemes, such as VBPR and BM3, show only limited gains. (2) The text modality alone achieves performance comparable to the full multimodal setting in most cases, whereas the image modality alone does not. These results offer foundational insights and practical guidance for the MMRec community. We will release our code and datasets to facilitate future research.

Problem

• Investigates if multimodal embeddings improve recommendation performance.
• Examines individual text and visual modalities' impact on recommendations.
• Evaluates 14 MMRec models to validate multimodal benefits.

Innovation

• Modality knockout strategy isolates individual modality effects
• Evaluates 14 state-of-the-art MMRec models comprehensively
• Text modality alone matches full multimodal performance