Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

2023-08-30

ACM Trans. Inf. Syst.

AI Summary

To address the limitation of static multimodal fusion in middle-school micro-video recommendation—its inability to capture inter-video modality relationship discrepancies—this paper proposes MetaMMF, a meta-learning-based dynamic multimodal fusion framework. Methodologically, MetaMMF treats multimodal fusion for each video as an individual meta-task and employs meta-learning to generate video-specific fusion functions; it further adopts CP tensor decomposition to enhance parameter efficiency and training stability. While implicitly incorporating graph neural network principles (e.g., akin to MMGCN), MetaMMF avoids explicit graph construction. Extensive experiments on three benchmark datasets demonstrate that MetaMMF consistently outperforms state-of-the-art models—including MMGCN, LATTICE, and InvRL—achieving superior recommendation accuracy and computational efficiency. The source code is publicly released, empirically validating the dual advantages of dynamic fusion in both performance and efficiency.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this article, we develop a novel meta-learning-based multimodal fusion framework called Meta Multimodal Fusion (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at https://github.com/hanliu95/MetaMMF.

Problem

• Multimodal Information Fusion
• Video Recommendation System
• Middle School Students

Innovation

• MetaMMF
• Personalized Multimodal Fusion
• Efficient Training